2026-02-22T21:24:12.675 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T21:24:12.691 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T21:24:12.775 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/verify/{centos_latest ceph clusters/fixed-4 d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64029' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 160 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 240 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - OSD bench result sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 stop-daemons-timeout: 1200 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial051.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZkZJqwfqN94NjV/kp3TXRAPlqgadxZzATQonuCpSVvpKHwer3AYOzAYa2UDQuY/HywWJlQjINSLKaenAe4MXs= trial068.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKr+hv4I49yU6HT5IHdc3OZKdUXBoqwRpYG7ScRKM+2nhaaZgo0worr5sH10dw141BaER3bmx5aSUpqx1euzJtA= trial125.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMG6J4x4AMod3GhuIGYLym1ttPAQL3O6gC+eNb4hFz6B0PjZ7XjHcLGK2phheE00Xp9JhqNtwVoUXzwPym9/3/0= trial140.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHixoq3LGQW2TB0fUQ+hOlmy4Wu0sLaq6pcu+ALrX0+2zRspvWPcV5BpQq/s5LQFNUrxG5irsEfRUB0vJgJ96Nk= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - '-- Don''t run this test on tiny clusters if py_len(yaml.roles) < 4 then reject() end ' teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T21:24:12.775 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T21:24:12.775 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T21:24:12.775 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T21:24:12.777 INFO:teuthology.task.internal:Checking packages... 2026-02-22T21:24:12.777 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T21:24:12.777 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T21:24:12.777 INFO:teuthology.packaging:ref: None 2026-02-22T21:24:12.777 INFO:teuthology.packaging:tag: None 2026-02-22T21:24:12.777 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T21:24:12.777 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:24:12.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T21:24:12.893 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689.gf90f4eae 2026-02-22T21:24:12.895 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T21:24:12.906 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T21:24:12.907 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T21:24:12.920 INFO:teuthology.task.internal:Saving configuration 2026-02-22T21:24:12.930 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T21:24:12.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T21:24:13.027 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial051.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 21:20:42.333009', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZkZJqwfqN94NjV/kp3TXRAPlqgadxZzATQonuCpSVvpKHwer3AYOzAYa2UDQuY/HywWJlQjINSLKaenAe4MXs='} 2026-02-22T21:24:13.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial068.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 21:20:42.332186', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKr+hv4I49yU6HT5IHdc3OZKdUXBoqwRpYG7ScRKM+2nhaaZgo0worr5sH10dw141BaER3bmx5aSUpqx1euzJtA='} 2026-02-22T21:24:13.137 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial125.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 21:20:42.331631', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMG6J4x4AMod3GhuIGYLym1ttPAQL3O6gC+eNb4hFz6B0PjZ7XjHcLGK2phheE00Xp9JhqNtwVoUXzwPym9/3/0='} 2026-02-22T21:24:13.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial140.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 21:20:42.332606', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHixoq3LGQW2TB0fUQ+hOlmy4Wu0sLaq6pcu+ALrX0+2zRspvWPcV5BpQq/s5LQFNUrxG5irsEfRUB0vJgJ96Nk='} 2026-02-22T21:24:13.194 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T21:24:13.205 INFO:teuthology.task.internal:roles: ubuntu@trial051.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2026-02-22T21:24:13.206 INFO:teuthology.task.internal:roles: ubuntu@trial068.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2026-02-22T21:24:13.206 INFO:teuthology.task.internal:roles: ubuntu@trial125.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2026-02-22T21:24:13.206 INFO:teuthology.task.internal:roles: ubuntu@trial140.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2026-02-22T21:24:13.206 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T21:24:13.460 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x788076708ea0>, signals=[15]) 2026-02-22T21:24:13.460 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T21:24:13.472 INFO:teuthology.task.internal:Opening connections... 2026-02-22T21:24:13.472 DEBUG:teuthology.task.internal:connecting to ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:24:13.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:24:13.534 DEBUG:teuthology.task.internal:connecting to ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:24:13.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:24:13.597 DEBUG:teuthology.task.internal:connecting to ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:24:13.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:24:13.656 DEBUG:teuthology.task.internal:connecting to ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:24:13.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:24:13.721 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T21:24:13.733 DEBUG:teuthology.orchestra.run.trial051:> uname -m 2026-02-22T21:24:13.748 INFO:teuthology.orchestra.run.trial051.stdout:x86_64 2026-02-22T21:24:13.748 DEBUG:teuthology.orchestra.run.trial051:> cat /etc/os-release 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:NAME="CentOS Stream" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:VERSION="9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:ID="centos" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:ID_LIKE="rhel fedora" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:VERSION_ID="9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:PLATFORM_ID="platform:el9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:ANSI_COLOR="0;31" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:LOGO="fedora-logo-icon" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:HOME_URL="https://centos.org/" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-22T21:24:13.804 INFO:teuthology.orchestra.run.trial051.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-22T21:24:13.805 INFO:teuthology.lock.ops:Updating trial051.front.sepia.ceph.com on lock server 2026-02-22T21:24:13.879 DEBUG:teuthology.orchestra.run.trial068:> uname -m 2026-02-22T21:24:13.894 INFO:teuthology.orchestra.run.trial068.stdout:x86_64 2026-02-22T21:24:13.895 DEBUG:teuthology.orchestra.run.trial068:> cat /etc/os-release 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:NAME="CentOS Stream" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:VERSION="9" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:ID="centos" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:ID_LIKE="rhel fedora" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:VERSION_ID="9" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:PLATFORM_ID="platform:el9" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:ANSI_COLOR="0;31" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:LOGO="fedora-logo-icon" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:HOME_URL="https://centos.org/" 2026-02-22T21:24:13.950 INFO:teuthology.orchestra.run.trial068.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-22T21:24:13.951 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-22T21:24:13.951 INFO:teuthology.orchestra.run.trial068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-22T21:24:13.951 INFO:teuthology.lock.ops:Updating trial068.front.sepia.ceph.com on lock server 2026-02-22T21:24:14.034 DEBUG:teuthology.orchestra.run.trial125:> uname -m 2026-02-22T21:24:14.049 INFO:teuthology.orchestra.run.trial125.stdout:x86_64 2026-02-22T21:24:14.050 DEBUG:teuthology.orchestra.run.trial125:> cat /etc/os-release 2026-02-22T21:24:14.104 INFO:teuthology.orchestra.run.trial125.stdout:NAME="CentOS Stream" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:VERSION="9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:ID="centos" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:ID_LIKE="rhel fedora" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:VERSION_ID="9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:PLATFORM_ID="platform:el9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:ANSI_COLOR="0;31" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:LOGO="fedora-logo-icon" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:HOME_URL="https://centos.org/" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-22T21:24:14.105 INFO:teuthology.orchestra.run.trial125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-22T21:24:14.105 INFO:teuthology.lock.ops:Updating trial125.front.sepia.ceph.com on lock server 2026-02-22T21:24:14.166 DEBUG:teuthology.orchestra.run.trial140:> uname -m 2026-02-22T21:24:14.181 INFO:teuthology.orchestra.run.trial140.stdout:x86_64 2026-02-22T21:24:14.181 DEBUG:teuthology.orchestra.run.trial140:> cat /etc/os-release 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:NAME="CentOS Stream" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:VERSION="9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:ID="centos" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:ID_LIKE="rhel fedora" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:VERSION_ID="9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:PLATFORM_ID="platform:el9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:ANSI_COLOR="0;31" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:LOGO="fedora-logo-icon" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:HOME_URL="https://centos.org/" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-22T21:24:14.237 INFO:teuthology.orchestra.run.trial140.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-22T21:24:14.238 INFO:teuthology.lock.ops:Updating trial140.front.sepia.ceph.com on lock server 2026-02-22T21:24:14.315 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T21:24:14.327 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T21:24:14.343 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T21:24:14.344 DEBUG:teuthology.orchestra.run.trial051:> test '!' -e /home/ubuntu/cephtest 2026-02-22T21:24:14.345 DEBUG:teuthology.orchestra.run.trial068:> test '!' -e /home/ubuntu/cephtest 2026-02-22T21:24:14.346 DEBUG:teuthology.orchestra.run.trial125:> test '!' -e /home/ubuntu/cephtest 2026-02-22T21:24:14.348 DEBUG:teuthology.orchestra.run.trial140:> test '!' -e /home/ubuntu/cephtest 2026-02-22T21:24:14.362 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T21:24:14.373 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T21:24:14.373 DEBUG:teuthology.orchestra.run.trial051:> test -z $(ls -A /var/lib/ceph) 2026-02-22T21:24:14.400 DEBUG:teuthology.orchestra.run.trial068:> test -z $(ls -A /var/lib/ceph) 2026-02-22T21:24:14.402 DEBUG:teuthology.orchestra.run.trial125:> test -z $(ls -A /var/lib/ceph) 2026-02-22T21:24:14.403 DEBUG:teuthology.orchestra.run.trial140:> test -z $(ls -A /var/lib/ceph) 2026-02-22T21:24:14.420 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T21:24:14.691 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T21:24:14.709 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T21:24:14.709 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'mon.b': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'mon.c': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T21:24:14.710 DEBUG:teuthology.orchestra.run.trial051:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T21:24:14.710 DEBUG:teuthology.orchestra.run.trial068:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T21:24:14.710 DEBUG:teuthology.orchestra.run.trial125:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T21:24:14.710 DEBUG:teuthology.orchestra.run.trial140:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T21:24:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:14.725 DEBUG:teuthology.orchestra.run.trial051:> uname -r 2026-02-22T21:24:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:14.725 DEBUG:teuthology.orchestra.run.trial125:> uname -r 2026-02-22T21:24:14.725 DEBUG:teuthology.orchestra.run.trial068:> uname -r 2026-02-22T21:24:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:14.726 DEBUG:teuthology.orchestra.run.trial140:> uname -r 2026-02-22T21:24:14.780 INFO:teuthology.orchestra.run.trial068.stdout:5.14.0-681.el9.x86_64 2026-02-22T21:24:14.780 INFO:teuthology.task.kernel:Running kernel on trial068: 5.14.0-681.el9.x86_64 2026-02-22T21:24:14.780 DEBUG:teuthology.orchestra.run.trial068:> sudo yum install -y kernel 2026-02-22T21:24:14.781 INFO:teuthology.orchestra.run.trial125.stdout:5.14.0-681.el9.x86_64 2026-02-22T21:24:14.781 INFO:teuthology.orchestra.run.trial051.stdout:5.14.0-681.el9.x86_64 2026-02-22T21:24:14.781 INFO:teuthology.task.kernel:Running kernel on trial125: 5.14.0-681.el9.x86_64 2026-02-22T21:24:14.781 DEBUG:teuthology.orchestra.run.trial125:> sudo yum install -y kernel 2026-02-22T21:24:14.781 INFO:teuthology.task.kernel:Running kernel on trial051: 5.14.0-681.el9.x86_64 2026-02-22T21:24:14.781 DEBUG:teuthology.orchestra.run.trial051:> sudo yum install -y kernel 2026-02-22T21:24:14.781 INFO:teuthology.orchestra.run.trial140.stdout:5.14.0-681.el9.x86_64 2026-02-22T21:24:14.782 INFO:teuthology.task.kernel:Running kernel on trial140: 5.14.0-681.el9.x86_64 2026-02-22T21:24:14.782 DEBUG:teuthology.orchestra.run.trial140:> sudo yum install -y kernel 2026-02-22T21:24:15.924 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-22T21:24:16.606 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.9 MB 00:01 2026-02-22T21:24:16.663 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.9 MB 00:01 2026-02-22T21:24:16.938 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.9 MB 00:01 2026-02-22T21:24:19.137 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-22T21:24:19.376 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 27 MB 00:02 2026-02-22T21:24:20.230 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-22T21:24:20.357 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 27 MB 00:03 2026-02-22T21:24:22.825 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 7.8 MB 00:01 2026-02-22T21:24:22.976 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - CRB 29 MB/s | 7.8 MB 00:00 2026-02-22T21:24:23.059 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 7.8 MB 00:01 2026-02-22T21:24:23.872 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - Extras packages 224 kB/s | 20 kB 00:00 2026-02-22T21:24:23.880 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 20 kB 00:00 2026-02-22T21:24:23.957 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 7.8 MB 00:01 2026-02-22T21:24:24.663 INFO:teuthology.orchestra.run.trial051.stdout:Extra Packages for Enterprise Linux 28 MB/s | 20 MB 00:00 2026-02-22T21:24:24.707 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:01 2026-02-22T21:24:25.151 INFO:teuthology.orchestra.run.trial140.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-22T21:24:25.228 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 20 kB 00:00 2026-02-22T21:24:25.649 INFO:teuthology.orchestra.run.trial125.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-22T21:24:25.960 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 30 MB/s | 20 MB 00:00 2026-02-22T21:24:27.863 INFO:teuthology.orchestra.run.trial051.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-22T21:24:28.345 INFO:teuthology.orchestra.run.trial140.stdout:lab-extras 2.9 MB/s | 50 kB 00:00 2026-02-22T21:24:28.799 INFO:teuthology.orchestra.run.trial125.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-22T21:24:28.876 INFO:teuthology.orchestra.run.trial051.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-22T21:24:28.876 INFO:teuthology.orchestra.run.trial051.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-22T21:24:28.895 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:24:28.896 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:24:28.896 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:24:28.919 DEBUG:teuthology.orchestra.run.trial051:> echo no | sudo yum reinstall kernel || true 2026-02-22T21:24:29.139 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-22T21:24:29.219 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 09:24:27 PM UTC. 2026-02-22T21:24:29.303 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:24:29.305 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.305 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:29.305 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.305 INFO:teuthology.orchestra.run.trial051.stdout:Reinstalling: 2026-02-22T21:24:29.306 INFO:teuthology.orchestra.run.trial051.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:29.306 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:29.306 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:24:29.306 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.306 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:29.307 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 803 k 2026-02-22T21:24:29.307 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 0 2026-02-22T21:24:29.307 INFO:teuthology.orchestra.run.trial051.stderr:Operation aborted. 2026-02-22T21:24:29.324 INFO:teuthology.orchestra.run.trial051.stdout:Is this ok [y/N]: 2026-02-22T21:24:29.325 DEBUG:teuthology.orchestra.run.trial051:> sudo yum reinstall -y kernel || true 2026-02-22T21:24:29.357 INFO:teuthology.orchestra.run.trial140.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-22T21:24:29.357 INFO:teuthology.orchestra.run.trial140.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-22T21:24:29.376 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:24:29.377 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:24:29.377 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:24:29.400 DEBUG:teuthology.orchestra.run.trial140:> echo no | sudo yum reinstall kernel || true 2026-02-22T21:24:29.624 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 09:24:27 PM UTC. 2026-02-22T21:24:29.700 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:01 ago on Sun 22 Feb 2026 09:24:28 PM UTC. 2026-02-22T21:24:29.704 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout:Reinstalling: 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:24:29.705 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:29.706 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 803 k 2026-02-22T21:24:29.706 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 0 2026-02-22T21:24:29.706 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:24:29.780 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:24:29.781 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:29.781 INFO:teuthology.orchestra.run.trial140.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:29.781 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout:Reinstalling: 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 803 k 2026-02-22T21:24:29.782 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 0 2026-02-22T21:24:29.783 INFO:teuthology.orchestra.run.trial140.stderr:Operation aborted. 2026-02-22T21:24:29.801 INFO:teuthology.orchestra.run.trial140.stdout:Is this ok [y/N]: 2026-02-22T21:24:29.802 DEBUG:teuthology.orchestra.run.trial140:> sudo yum reinstall -y kernel || true 2026-02-22T21:24:29.812 INFO:teuthology.orchestra.run.trial125.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-22T21:24:29.812 INFO:teuthology.orchestra.run.trial125.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-22T21:24:29.831 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:24:29.832 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:24:29.832 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:24:29.856 DEBUG:teuthology.orchestra.run.trial125:> echo no | sudo yum reinstall kernel || true 2026-02-22T21:24:30.102 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 09:24:28 PM UTC. 2026-02-22T21:24:30.146 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-22T21:24:30.146 INFO:teuthology.orchestra.run.trial068.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-22T21:24:30.160 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 09:24:28 PM UTC. 2026-02-22T21:24:30.165 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:24:30.165 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:24:30.165 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:24:30.182 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:24:30.182 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:30.182 INFO:teuthology.orchestra.run.trial140.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:30.182 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:Reinstalling: 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 803 k 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 0 2026-02-22T21:24:30.183 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:24:30.189 DEBUG:teuthology.orchestra.run.trial068:> echo no | sudo yum reinstall kernel || true 2026-02-22T21:24:30.241 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:24:30.241 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:Reinstalling: 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 803 k 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 0 2026-02-22T21:24:30.242 INFO:teuthology.orchestra.run.trial125.stderr:Operation aborted. 2026-02-22T21:24:30.262 INFO:teuthology.orchestra.run.trial125.stdout:Is this ok [y/N]: 2026-02-22T21:24:30.263 DEBUG:teuthology.orchestra.run.trial125:> sudo yum reinstall -y kernel || true 2026-02-22T21:24:30.490 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:01 ago on Sun 22 Feb 2026 09:24:29 PM UTC. 2026-02-22T21:24:30.565 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 09:24:28 PM UTC. 2026-02-22T21:24:30.570 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:24:30.571 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.571 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:30.571 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.571 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalling: 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 803 k 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 0 2026-02-22T21:24:30.572 INFO:teuthology.orchestra.run.trial068.stderr:Operation aborted. 2026-02-22T21:24:30.591 INFO:teuthology.orchestra.run.trial068.stdout:Is this ok [y/N]: 2026-02-22T21:24:30.592 DEBUG:teuthology.orchestra.run.trial068:> sudo yum reinstall -y kernel || true 2026-02-22T21:24:30.622 INFO:teuthology.orchestra.run.trial140.stdout:kernel-5.14.0-681.el9.x86_64.rpm 5.7 MB/s | 803 kB 00:00 2026-02-22T21:24:30.622 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:24:30.622 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.8 MB/s | 803 kB 00:00 2026-02-22T21:24:30.632 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:24:30.645 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:24:30.645 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.645 INFO:teuthology.orchestra.run.trial125.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:Reinstalling: 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 803 k 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 0 2026-02-22T21:24:30.646 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:24:30.666 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:24:30.667 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:24:30.671 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:24:30.671 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:24:30.695 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:24:30.699 INFO:teuthology.orchestra.run.trial140.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:30.699 INFO:teuthology.orchestra.run.trial140.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:30.807 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:30.807 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:30.862 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:30.863 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:30.863 INFO:teuthology.orchestra.run.trial140.stdout:Reinstalled: 2026-02-22T21:24:30.863 INFO:teuthology.orchestra.run.trial140.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:30.863 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:24:30.863 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:24:30.890 DEBUG:teuthology.orchestra.run.trial140:> rpm -q kernel | sort -rV | head -n 1 2026-02-22T21:24:30.891 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:01 ago on Sun 22 Feb 2026 09:24:29 PM UTC. 2026-02-22T21:24:30.915 INFO:teuthology.orchestra.run.trial140.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:30.915 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-22T21:24:30.916 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T21:24:30.916 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-22T21:24:30.916 DEBUG:teuthology.orchestra.run.trial140:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T21:24:30.970 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:24:30.971 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.971 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-22T21:24:30.971 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalling: 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 803 k 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 0 2026-02-22T21:24:30.972 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:24:30.979 INFO:teuthology.orchestra.run.trial140.stdout:ttyS1 2026-02-22T21:24:31.003 DEBUG:teuthology.parallel:result is None 2026-02-22T21:24:31.160 INFO:teuthology.orchestra.run.trial051.stdout:kernel-5.14.0-681.el9.x86_64.rpm 576 kB/s | 803 kB 00:01 2026-02-22T21:24:31.161 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:24:31.161 INFO:teuthology.orchestra.run.trial051.stdout:Total 551 kB/s | 803 kB 00:01 2026-02-22T21:24:31.171 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:24:31.205 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:24:31.205 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:24:31.209 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:24:31.209 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:24:31.229 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-681.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2026-02-22T21:24:31.229 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:24:31.229 INFO:teuthology.orchestra.run.trial068.stdout:Total 3.0 MB/s | 803 kB 00:00 2026-02-22T21:24:31.233 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:24:31.236 INFO:teuthology.orchestra.run.trial051.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.237 INFO:teuthology.orchestra.run.trial051.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.239 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:24:31.272 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:24:31.272 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:24:31.277 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:24:31.277 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:24:31.300 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:24:31.304 INFO:teuthology.orchestra.run.trial068.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.305 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.345 INFO:teuthology.orchestra.run.trial125.stdout:kernel-5.14.0-681.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2026-02-22T21:24:31.345 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:24:31.345 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.1 MB/s | 803 kB 00:00 2026-02-22T21:24:31.349 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.349 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.355 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:24:31.389 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:24:31.389 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:24:31.394 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:24:31.394 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout:Reinstalled: 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:24:31.401 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:24:31.413 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.413 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.418 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:24:31.421 INFO:teuthology.orchestra.run.trial125.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.422 INFO:teuthology.orchestra.run.trial125.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.428 DEBUG:teuthology.orchestra.run.trial051:> rpm -q kernel | sort -rV | head -n 1 2026-02-22T21:24:31.452 INFO:teuthology.orchestra.run.trial051.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.453 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-22T21:24:31.453 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T21:24:31.453 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T21:24:31.454 DEBUG:teuthology.orchestra.run.trial051:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout:Reinstalled: 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:24:31.470 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:24:31.497 DEBUG:teuthology.orchestra.run.trial068:> rpm -q kernel | sort -rV | head -n 1 2026-02-22T21:24:31.516 INFO:teuthology.orchestra.run.trial051.stdout:ttyS1 2026-02-22T21:24:31.521 INFO:teuthology.orchestra.run.trial068.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.522 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-22T21:24:31.522 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T21:24:31.522 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2026-02-22T21:24:31.523 DEBUG:teuthology.orchestra.run.trial068:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T21:24:31.533 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.533 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T21:24:31.547 DEBUG:teuthology.parallel:result is None 2026-02-22T21:24:31.586 INFO:teuthology.orchestra.run.trial068.stdout:ttyS1 2026-02-22T21:24:31.588 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T21:24:31.588 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:31.588 INFO:teuthology.orchestra.run.trial125.stdout:Reinstalled: 2026-02-22T21:24:31.588 INFO:teuthology.orchestra.run.trial125.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.588 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:24:31.589 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:24:31.613 DEBUG:teuthology.parallel:result is None 2026-02-22T21:24:31.615 DEBUG:teuthology.orchestra.run.trial125:> rpm -q kernel | sort -rV | head -n 1 2026-02-22T21:24:31.680 INFO:teuthology.orchestra.run.trial125.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-22T21:24:31.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-22T21:24:31.681 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T21:24:31.681 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2026-02-22T21:24:31.681 DEBUG:teuthology.orchestra.run.trial125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T21:24:31.744 INFO:teuthology.orchestra.run.trial125.stdout:ttyS1 2026-02-22T21:24:31.768 DEBUG:teuthology.parallel:result is None 2026-02-22T21:24:31.768 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T21:24:31.775 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T21:24:31.776 DEBUG:teuthology.orchestra.run.trial051:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T21:24:31.779 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T21:24:31.782 DEBUG:teuthology.orchestra.run.trial125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T21:24:31.810 DEBUG:teuthology.orchestra.run.trial140:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T21:24:31.826 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T21:24:31.838 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T21:24:31.851 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T21:24:31.852 DEBUG:teuthology.orchestra.run.trial051:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T21:24:31.854 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T21:24:31.856 DEBUG:teuthology.orchestra.run.trial125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T21:24:31.866 DEBUG:teuthology.orchestra.run.trial140:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T21:24:31.891 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T21:24:31.905 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T21:24:31.905 DEBUG:teuthology.orchestra.run.trial051:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T21:24:31.910 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T21:24:31.914 DEBUG:teuthology.orchestra.run.trial125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T21:24:31.922 DEBUG:teuthology.orchestra.run.trial140:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T21:24:31.931 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.935 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.940 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.944 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.944 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.948 INFO:teuthology.orchestra.run.trial140.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.953 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.957 INFO:teuthology.orchestra.run.trial140.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:24:31.958 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T21:24:31.969 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T21:24:31.970 DEBUG:teuthology.orchestra.run.trial051:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T21:24:31.984 DEBUG:teuthology.orchestra.run.trial068:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T21:24:31.987 DEBUG:teuthology.orchestra.run.trial125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T21:24:31.997 DEBUG:teuthology.orchestra.run.trial140:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T21:24:32.022 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T21:24:32.036 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T21:24:32.037 DEBUG:teuthology.orchestra.run.trial051:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T21:24:32.048 DEBUG:teuthology.orchestra.run.trial068:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T21:24:32.051 DEBUG:teuthology.orchestra.run.trial125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T21:24:32.060 DEBUG:teuthology.orchestra.run.trial140:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T21:24:32.078 DEBUG:teuthology.orchestra.run.trial051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.178 DEBUG:teuthology.orchestra.run.trial051:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.201 DEBUG:teuthology.orchestra.run.trial051:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.334 DEBUG:teuthology.orchestra.run.trial051:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.357 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:24:32.357 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T21:24:32.421 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.495 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.519 DEBUG:teuthology.orchestra.run.trial068:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.639 DEBUG:teuthology.orchestra.run.trial068:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.661 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:24:32.661 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T21:24:32.726 DEBUG:teuthology.orchestra.run.trial125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.802 DEBUG:teuthology.orchestra.run.trial125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:32.825 DEBUG:teuthology.orchestra.run.trial125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.956 DEBUG:teuthology.orchestra.run.trial125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:32.979 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:24:32.979 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T21:24:33.043 DEBUG:teuthology.orchestra.run.trial140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:33.146 DEBUG:teuthology.orchestra.run.trial140:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:24:33.170 DEBUG:teuthology.orchestra.run.trial140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:33.286 DEBUG:teuthology.orchestra.run.trial140:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:24:33.309 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:24:33.309 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T21:24:33.372 DEBUG:teuthology.orchestra.run.trial051:> sudo service rsyslog restart 2026-02-22T21:24:33.374 DEBUG:teuthology.orchestra.run.trial068:> sudo service rsyslog restart 2026-02-22T21:24:33.376 DEBUG:teuthology.orchestra.run.trial125:> sudo service rsyslog restart 2026-02-22T21:24:33.377 DEBUG:teuthology.orchestra.run.trial140:> sudo service rsyslog restart 2026-02-22T21:24:33.400 INFO:teuthology.orchestra.run.trial051.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:24:33.401 INFO:teuthology.orchestra.run.trial068.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:24:33.403 INFO:teuthology.orchestra.run.trial125.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:24:33.441 INFO:teuthology.orchestra.run.trial140.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:24:33.919 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T21:24:33.931 INFO:teuthology.task.internal:Starting timer... 2026-02-22T21:24:33.931 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T21:24:33.948 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T21:24:33.961 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2026-02-22T21:24:33.961 DEBUG:teuthology.orchestra.run.trial051:> sudo service auditd rotate 2026-02-22T21:24:33.989 INFO:teuthology.orchestra.run.trial051.stdout:Rotating logs: 2026-02-22T21:24:33.991 DEBUG:teuthology.orchestra.run.trial068:> sudo service auditd rotate 2026-02-22T21:24:34.020 INFO:teuthology.orchestra.run.trial068.stdout:Rotating logs: 2026-02-22T21:24:34.022 DEBUG:teuthology.orchestra.run.trial125:> sudo service auditd rotate 2026-02-22T21:24:34.051 INFO:teuthology.orchestra.run.trial125.stdout:Rotating logs: 2026-02-22T21:24:34.053 DEBUG:teuthology.orchestra.run.trial140:> sudo service auditd rotate 2026-02-22T21:24:34.083 INFO:teuthology.orchestra.run.trial140.stdout:Rotating logs: 2026-02-22T21:24:34.085 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T21:24:34.085 DEBUG:teuthology.orchestra.run.trial051:> /usr/sbin/getenforce 2026-02-22T21:24:34.102 INFO:teuthology.orchestra.run.trial051.stdout:Permissive 2026-02-22T21:24:34.102 DEBUG:teuthology.orchestra.run.trial068:> /usr/sbin/getenforce 2026-02-22T21:24:34.118 INFO:teuthology.orchestra.run.trial068.stdout:Permissive 2026-02-22T21:24:34.118 DEBUG:teuthology.orchestra.run.trial125:> /usr/sbin/getenforce 2026-02-22T21:24:34.134 INFO:teuthology.orchestra.run.trial125.stdout:Permissive 2026-02-22T21:24:34.135 DEBUG:teuthology.orchestra.run.trial140:> /usr/sbin/getenforce 2026-02-22T21:24:34.152 INFO:teuthology.orchestra.run.trial140.stdout:Permissive 2026-02-22T21:24:34.152 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial051.front.sepia.ceph.com': 'permissive', 'ubuntu@trial068.front.sepia.ceph.com': 'permissive', 'ubuntu@trial125.front.sepia.ceph.com': 'permissive', 'ubuntu@trial140.front.sepia.ceph.com': 'permissive'} 2026-02-22T21:24:34.153 DEBUG:teuthology.orchestra.run.trial051:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:24:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:34.179 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:24:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:34.203 DEBUG:teuthology.orchestra.run.trial125:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:24:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:34.227 DEBUG:teuthology.orchestra.run.trial140:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:24:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:24:34.252 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T21:24:34.252 DEBUG:teuthology.orchestra.run.trial051:> sudo /usr/sbin/setenforce permissive 2026-02-22T21:24:34.275 DEBUG:teuthology.orchestra.run.trial068:> sudo /usr/sbin/setenforce permissive 2026-02-22T21:24:34.300 DEBUG:teuthology.orchestra.run.trial125:> sudo /usr/sbin/setenforce permissive 2026-02-22T21:24:34.324 DEBUG:teuthology.orchestra.run.trial140:> sudo /usr/sbin/setenforce permissive 2026-02-22T21:24:34.349 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T21:24:34.371 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T21:24:34.382 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T21:24:34.384 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial051.front.sepia.ceph.com,trial068.front.sepia.ceph.com,trial125.front.sepia.ceph.com,trial140.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T21:27:14.105 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial051.front.sepia.ceph.com'), Remote(name='ubuntu@trial068.front.sepia.ceph.com'), Remote(name='ubuntu@trial125.front.sepia.ceph.com'), Remote(name='ubuntu@trial140.front.sepia.ceph.com')] 2026-02-22T21:27:14.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-22T21:27:14.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial051.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:27:14.200 DEBUG:teuthology.orchestra.run.trial051:> true 2026-02-22T21:27:14.280 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial051.front.sepia.ceph.com' 2026-02-22T21:27:14.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-22T21:27:14.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial068.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:27:14.368 DEBUG:teuthology.orchestra.run.trial068:> true 2026-02-22T21:27:14.443 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial068.front.sepia.ceph.com' 2026-02-22T21:27:14.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-22T21:27:14.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:27:14.505 DEBUG:teuthology.orchestra.run.trial125:> true 2026-02-22T21:27:14.587 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial125.front.sepia.ceph.com' 2026-02-22T21:27:14.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial140.front.sepia.ceph.com' 2026-02-22T21:27:14.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:27:14.721 DEBUG:teuthology.orchestra.run.trial140:> true 2026-02-22T21:27:14.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial140.front.sepia.ceph.com' 2026-02-22T21:27:14.803 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T21:27:14.818 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T21:27:14.818 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T21:27:14.818 DEBUG:teuthology.orchestra.run.trial051:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:27:14.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T21:27:14.820 DEBUG:teuthology.orchestra.run.trial068:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:27:14.823 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T21:27:14.823 DEBUG:teuthology.orchestra.run.trial125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:27:14.825 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T21:27:14.825 DEBUG:teuthology.orchestra.run.trial140:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:27:14.846 INFO:teuthology.orchestra.run.trial051.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-22T21:27:14.849 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-22T21:27:14.850 INFO:teuthology.orchestra.run.trial125.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-22T21:27:14.858 INFO:teuthology.orchestra.run.trial051.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-22T21:27:14.861 INFO:teuthology.orchestra.run.trial068.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-22T21:27:14.862 INFO:teuthology.orchestra.run.trial125.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-22T21:27:14.870 INFO:teuthology.orchestra.run.trial140.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-22T21:27:14.878 INFO:teuthology.orchestra.run.trial051.stderr:sudo: ntpd: command not found 2026-02-22T21:27:14.881 INFO:teuthology.orchestra.run.trial140.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-22T21:27:14.882 INFO:teuthology.orchestra.run.trial068.stderr:sudo: ntpd: command not found 2026-02-22T21:27:14.882 INFO:teuthology.orchestra.run.trial125.stderr:sudo: ntpd: command not found 2026-02-22T21:27:14.887 INFO:teuthology.orchestra.run.trial051.stdout:506 Cannot talk to daemon 2026-02-22T21:27:14.892 INFO:teuthology.orchestra.run.trial125.stdout:506 Cannot talk to daemon 2026-02-22T21:27:14.892 INFO:teuthology.orchestra.run.trial068.stdout:506 Cannot talk to daemon 2026-02-22T21:27:14.899 INFO:teuthology.orchestra.run.trial051.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-22T21:27:14.902 INFO:teuthology.orchestra.run.trial140.stderr:sudo: ntpd: command not found 2026-02-22T21:27:14.903 INFO:teuthology.orchestra.run.trial125.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-22T21:27:14.903 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-22T21:27:14.911 INFO:teuthology.orchestra.run.trial051.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-22T21:27:14.911 INFO:teuthology.orchestra.run.trial140.stdout:506 Cannot talk to daemon 2026-02-22T21:27:14.915 INFO:teuthology.orchestra.run.trial125.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-22T21:27:14.915 INFO:teuthology.orchestra.run.trial068.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-22T21:27:14.923 INFO:teuthology.orchestra.run.trial140.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-22T21:27:14.935 INFO:teuthology.orchestra.run.trial140.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-22T21:27:14.957 INFO:teuthology.orchestra.run.trial051.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:27:14.961 INFO:teuthology.orchestra.run.trial125.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:27:14.970 INFO:teuthology.orchestra.run.trial068.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:27:14.971 INFO:teuthology.orchestra.run.trial051.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:27:14.972 INFO:teuthology.orchestra.run.trial051.stdout:=============================================================================== 2026-02-22T21:27:14.972 INFO:teuthology.orchestra.run.trial051.stdout:^? 50.205.57.38 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.972 INFO:teuthology.orchestra.run.trial051.stdout:^? 73.185.182.209 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.972 INFO:teuthology.orchestra.run.trial051.stdout:^? 149.28.200.179.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.972 INFO:teuthology.orchestra.run.trial051.stdout:^? 96-19-94-82.cpe.sparklig> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.982 INFO:teuthology.orchestra.run.trial125.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:27:14.983 INFO:teuthology.orchestra.run.trial125.stdout:=============================================================================== 2026-02-22T21:27:14.983 INFO:teuthology.orchestra.run.trial125.stdout:^? 172-233-189-68.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.983 INFO:teuthology.orchestra.run.trial125.stdout:^? 44.190.5.123 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.983 INFO:teuthology.orchestra.run.trial125.stdout:^? white.web-ster.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.983 INFO:teuthology.orchestra.run.trial125.stdout:^? hou1.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.984 INFO:teuthology.orchestra.run.trial140.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:=============================================================================== 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:^? time.lmtlabs.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:^? ntp.speculation.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:^? 172-234-25-10.ip.linodeu> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.986 INFO:teuthology.orchestra.run.trial068.stdout:^? static.36.62.78.5.client> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:=============================================================================== 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:^? 172-232-15-202.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:^? vps5.drown.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:^? time.ecansol.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.998 INFO:teuthology.orchestra.run.trial140.stdout:^? 172-235-154-118.ip.linod> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T21:27:14.999 INFO:teuthology.run_tasks:Running task install... 2026-02-22T21:27:15.013 DEBUG:teuthology.task.install:project ceph 2026-02-22T21:27:15.013 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T21:27:15.013 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T21:27:15.013 INFO:teuthology.task.install:Using flavor: default 2026-02-22T21:27:15.021 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2026-02-22T21:27:15.022 INFO:teuthology.task.install:extra packages: [] 2026-02-22T21:27:15.022 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'tag': None, 'wait_for_package': False} 2026-02-22T21:27:15.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:27:15.025 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'tag': None, 'wait_for_package': False} 2026-02-22T21:27:15.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:27:15.027 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'tag': None, 'wait_for_package': False} 2026-02-22T21:27:15.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:27:15.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'tag': None, 'wait_for_package': False} 2026-02-22T21:27:15.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:27:15.305 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/ 2026-02-22T21:27:15.305 INFO:teuthology.task.install.rpm:Package version is 19.2.3-689.gf90f4eae 2026-02-22T21:27:15.308 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/ 2026-02-22T21:27:15.308 INFO:teuthology.task.install.rpm:Package version is 19.2.3-689.gf90f4eae 2026-02-22T21:27:15.310 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/ 2026-02-22T21:27:15.310 INFO:teuthology.task.install.rpm:Package version is 19.2.3-689.gf90f4eae 2026-02-22T21:27:15.313 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/ 2026-02-22T21:27:15.313 INFO:teuthology.task.install.rpm:Package version is 19.2.3-689.gf90f4eae 2026-02-22T21:27:15.544 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-22T21:27:15.544 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:27:15.544 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-22T21:27:15.546 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-22T21:27:15.546 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:27:15.546 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-22T21:27:15.548 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-22T21:27:15.548 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:27:15.548 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-22T21:27:15.552 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-22T21:27:15.552 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:27:15.553 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-22T21:27:15.572 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2026-02-22T21:27:15.572 DEBUG:teuthology.orchestra.run.trial140:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f90f4eaebb28b619cf76e80cbbe997656da69579/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-22T21:27:15.573 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2026-02-22T21:27:15.573 DEBUG:teuthology.orchestra.run.trial125:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f90f4eaebb28b619cf76e80cbbe997656da69579/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-22T21:27:15.573 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2026-02-22T21:27:15.573 DEBUG:teuthology.orchestra.run.trial068:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f90f4eaebb28b619cf76e80cbbe997656da69579/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-22T21:27:15.577 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2026-02-22T21:27:15.577 DEBUG:teuthology.orchestra.run.trial051:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f90f4eaebb28b619cf76e80cbbe997656da69579/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-22T21:27:15.636 DEBUG:teuthology.orchestra.run.trial125:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-22T21:27:15.637 DEBUG:teuthology.orchestra.run.trial140:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-22T21:27:15.637 DEBUG:teuthology.orchestra.run.trial068:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-22T21:27:15.641 DEBUG:teuthology.orchestra.run.trial051:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-22T21:27:15.708 DEBUG:teuthology.orchestra.run.trial125:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-22T21:27:15.715 DEBUG:teuthology.orchestra.run.trial068:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-22T21:27:15.720 DEBUG:teuthology.orchestra.run.trial140:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-22T21:27:15.732 DEBUG:teuthology.orchestra.run.trial051:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-22T21:27:15.772 INFO:teuthology.orchestra.run.trial125.stdout:check_obsoletes = 1 2026-02-22T21:27:15.773 DEBUG:teuthology.orchestra.run.trial125:> sudo yum clean all 2026-02-22T21:27:15.783 INFO:teuthology.orchestra.run.trial068.stdout:check_obsoletes = 1 2026-02-22T21:27:15.784 DEBUG:teuthology.orchestra.run.trial068:> sudo yum clean all 2026-02-22T21:27:15.790 INFO:teuthology.orchestra.run.trial140.stdout:check_obsoletes = 1 2026-02-22T21:27:15.791 DEBUG:teuthology.orchestra.run.trial140:> sudo yum clean all 2026-02-22T21:27:15.799 INFO:teuthology.orchestra.run.trial051.stdout:check_obsoletes = 1 2026-02-22T21:27:15.801 DEBUG:teuthology.orchestra.run.trial051:> sudo yum clean all 2026-02-22T21:27:15.931 INFO:teuthology.orchestra.run.trial125.stdout:41 files removed 2026-02-22T21:27:15.942 INFO:teuthology.orchestra.run.trial068.stdout:41 files removed 2026-02-22T21:27:15.946 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-radosgw 2026-02-22T21:27:15.949 INFO:teuthology.orchestra.run.trial140.stdout:41 files removed 2026-02-22T21:27:15.957 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-radosgw 2026-02-22T21:27:15.958 INFO:teuthology.orchestra.run.trial051.stdout:41 files removed 2026-02-22T21:27:15.963 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-radosgw 2026-02-22T21:27:15.973 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-radosgw 2026-02-22T21:27:16.218 INFO:teuthology.orchestra.run.trial125.stdout:ceph packages for x86_64 660 kB/s | 86 kB 00:00 2026-02-22T21:27:16.232 INFO:teuthology.orchestra.run.trial068.stdout:ceph packages for x86_64 648 kB/s | 86 kB 00:00 2026-02-22T21:27:16.232 INFO:teuthology.orchestra.run.trial140.stdout:ceph packages for x86_64 675 kB/s | 86 kB 00:00 2026-02-22T21:27:16.240 INFO:teuthology.orchestra.run.trial051.stdout:ceph packages for x86_64 679 kB/s | 86 kB 00:00 2026-02-22T21:27:16.348 INFO:teuthology.orchestra.run.trial125.stdout:ceph noarch packages 114 kB/s | 12 kB 00:00 2026-02-22T21:27:16.362 INFO:teuthology.orchestra.run.trial068.stdout:ceph noarch packages 113 kB/s | 12 kB 00:00 2026-02-22T21:27:16.366 INFO:teuthology.orchestra.run.trial140.stdout:ceph noarch packages 111 kB/s | 12 kB 00:00 2026-02-22T21:27:16.371 INFO:teuthology.orchestra.run.trial051.stdout:ceph noarch packages 113 kB/s | 12 kB 00:00 2026-02-22T21:27:16.474 INFO:teuthology.orchestra.run.trial125.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-22T21:27:16.489 INFO:teuthology.orchestra.run.trial068.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-22T21:27:16.490 INFO:teuthology.orchestra.run.trial140.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-22T21:27:16.496 INFO:teuthology.orchestra.run.trial051.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-22T21:27:17.181 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-22T21:27:17.197 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-22T21:27:17.338 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-22T21:27:18.208 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.9 MB 00:01 2026-02-22T21:27:19.964 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-22T21:27:20.077 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-22T21:27:20.503 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 27 MB 00:02 2026-02-22T21:27:21.255 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-22T21:27:23.425 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 7.8 MB 00:00 2026-02-22T21:27:24.027 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 7.8 MB 00:01 2026-02-22T21:27:24.607 INFO:teuthology.orchestra.run.trial140.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-22T21:27:24.624 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 7.8 MB 00:01 2026-02-22T21:27:24.658 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 7.8 MB 00:01 2026-02-22T21:27:25.493 INFO:teuthology.orchestra.run.trial125.stdout:CentOS Stream 9 - Extras packages 175 kB/s | 20 kB 00:00 2026-02-22T21:27:25.603 INFO:teuthology.orchestra.run.trial140.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-22T21:27:25.611 INFO:teuthology.orchestra.run.trial051.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 20 kB 00:00 2026-02-22T21:27:25.850 INFO:teuthology.orchestra.run.trial125.stdout:Extra Packages for Enterprise Linux 69 MB/s | 20 MB 00:00 2026-02-22T21:27:26.185 INFO:teuthology.orchestra.run.trial051.stdout:Extra Packages for Enterprise Linux 40 MB/s | 20 MB 00:00 2026-02-22T21:27:26.460 INFO:teuthology.orchestra.run.trial068.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-22T21:27:27.938 INFO:teuthology.orchestra.run.trial068.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2026-02-22T21:27:28.827 INFO:teuthology.orchestra.run.trial140.stdout:lab-extras 1.0 MB/s | 50 kB 00:00 2026-02-22T21:27:29.066 INFO:teuthology.orchestra.run.trial125.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-22T21:27:29.342 INFO:teuthology.orchestra.run.trial051.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-22T21:27:30.042 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:27:30.043 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:30.043 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repository Size 2026-02-22T21:27:30.043 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:30.043 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: ceph-radosgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout:Upgrading: 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: librados2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.4 M 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: librbd1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.2 M 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout:Installing dependencies: 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: ceph-base x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.5 M 2026-02-22T21:27:30.044 INFO:teuthology.orchestra.run.trial140.stdout: ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 22 M 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: ceph-selinux x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 25 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 999 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-22T21:27:30.045 INFO:teuthology.orchestra.run.trial140.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: libradosstriper1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 503 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: librgw2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.4 M 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-22T21:27:30.046 INFO:teuthology.orchestra.run.trial140.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-ceph-argparse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 45 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 142 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-cephfs x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 165 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-rados x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 322 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-rbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 303 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: python3-rgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 100 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-22T21:27:30.047 INFO:teuthology.orchestra.run.trial140.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout:Installing weak dependencies: 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:27:30.048 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:30.049 INFO:teuthology.orchestra.run.trial140.stdout:Install 33 Packages 2026-02-22T21:27:30.049 INFO:teuthology.orchestra.run.trial140.stdout:Upgrade 2 Packages 2026-02-22T21:27:30.049 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:30.050 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 63 M 2026-02-22T21:27:30.050 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:27:30.260 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repository Size 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: ceph-radosgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout:Upgrading: 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: librados2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.4 M 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: librbd1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.2 M 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout:Installing dependencies: 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: ceph-base x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.5 M 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 22 M 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: ceph-selinux x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 25 k 2026-02-22T21:27:30.261 INFO:teuthology.orchestra.run.trial125.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 999 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libradosstriper1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 503 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: librgw2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.4 M 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-ceph-argparse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 45 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 142 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-cephfs x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 165 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-rados x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 322 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-rbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 303 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: python3-rgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 100 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout:Installing weak dependencies: 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout:Install 33 Packages 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout:Upgrade 2 Packages 2026-02-22T21:27:30.262 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:30.263 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 63 M 2026-02-22T21:27:30.263 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:27:30.537 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:27:30.538 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout:Upgrading: 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: librados2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.4 M 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: librbd1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.2 M 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.5 M 2026-02-22T21:27:30.539 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 22 M 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: ceph-selinux x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 25 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 999 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: libradosstriper1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 503 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-22T21:27:30.540 INFO:teuthology.orchestra.run.trial051.stdout: librgw2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.4 M 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-argparse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 45 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 142 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 165 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 322 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 303 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 100 k 2026-02-22T21:27:30.541 INFO:teuthology.orchestra.run.trial051.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout:Installing weak dependencies: 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout:Install 33 Packages 2026-02-22T21:27:30.542 INFO:teuthology.orchestra.run.trial051.stdout:Upgrade 2 Packages 2026-02-22T21:27:30.543 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:30.543 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 63 M 2026-02-22T21:27:30.543 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:27:31.140 INFO:teuthology.orchestra.run.trial068.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-22T21:27:32.340 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout:Upgrading: 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout: librados2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.4 M 2026-02-22T21:27:32.341 INFO:teuthology.orchestra.run.trial068.stdout: librbd1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.2 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.5 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 22 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 25 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 999 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 503 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: librgw2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.4 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 45 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 142 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 165 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 322 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 303 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 100 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:Install 33 Packages 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout:Upgrade 2 Packages 2026-02-22T21:27:32.342 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:32.343 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 63 M 2026-02-22T21:27:32.343 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:27:32.721 INFO:teuthology.orchestra.run.trial125.stdout:(1/35): ceph-base-19.2.3-689.gf90f4eae.el9.x86_ 2.5 MB/s | 5.5 MB 00:02 2026-02-22T21:27:32.775 INFO:teuthology.orchestra.run.trial125.stdout:(2/35): ceph-selinux-19.2.3-689.gf90f4eae.el9.x 465 kB/s | 25 kB 00:00 2026-02-22T21:27:33.071 INFO:teuthology.orchestra.run.trial125.stdout:(3/35): libcephfs2-19.2.3-689.gf90f4eae.el9.x86 3.3 MB/s | 999 kB 00:00 2026-02-22T21:27:33.200 INFO:teuthology.orchestra.run.trial125.stdout:(4/35): libradosstriper1-19.2.3-689.gf90f4eae.e 3.8 MB/s | 503 kB 00:00 2026-02-22T21:27:33.493 INFO:teuthology.orchestra.run.trial051.stdout:(1/35): ceph-base-19.2.3-689.gf90f4eae.el9.x86_ 2.2 MB/s | 5.5 MB 00:02 2026-02-22T21:27:33.556 INFO:teuthology.orchestra.run.trial051.stdout:(2/35): ceph-selinux-19.2.3-689.gf90f4eae.el9.x 397 kB/s | 25 kB 00:00 2026-02-22T21:27:33.562 INFO:teuthology.orchestra.run.trial140.stdout:(1/35): ceph-base-19.2.3-689.gf90f4eae.el9.x86_ 1.8 MB/s | 5.5 MB 00:03 2026-02-22T21:27:33.629 INFO:teuthology.orchestra.run.trial140.stdout:(2/35): ceph-selinux-19.2.3-689.gf90f4eae.el9.x 379 kB/s | 25 kB 00:00 2026-02-22T21:27:33.965 INFO:teuthology.orchestra.run.trial051.stdout:(3/35): libcephfs2-19.2.3-689.gf90f4eae.el9.x86 2.4 MB/s | 999 kB 00:00 2026-02-22T21:27:34.075 INFO:teuthology.orchestra.run.trial140.stdout:(3/35): ceph-radosgw-19.2.3-689.gf90f4eae.el9.x 3.0 MB/s | 11 MB 00:03 2026-02-22T21:27:34.117 INFO:teuthology.orchestra.run.trial051.stdout:(4/35): libradosstriper1-19.2.3-689.gf90f4eae.e 3.2 MB/s | 503 kB 00:00 2026-02-22T21:27:34.181 INFO:teuthology.orchestra.run.trial140.stdout:(4/35): libcephfs2-19.2.3-689.gf90f4eae.el9.x86 1.8 MB/s | 999 kB 00:00 2026-02-22T21:27:34.183 INFO:teuthology.orchestra.run.trial140.stdout:(5/35): libradosstriper1-19.2.3-689.gf90f4eae.e 4.5 MB/s | 503 kB 00:00 2026-02-22T21:27:34.237 INFO:teuthology.orchestra.run.trial140.stdout:(6/35): python3-ceph-argparse-19.2.3-689.gf90f4 856 kB/s | 45 kB 00:00 2026-02-22T21:27:34.291 INFO:teuthology.orchestra.run.trial140.stdout:(7/35): python3-ceph-common-19.2.3-689.gf90f4ea 2.5 MB/s | 142 kB 00:00 2026-02-22T21:27:34.344 INFO:teuthology.orchestra.run.trial051.stdout:(5/35): ceph-radosgw-19.2.3-689.gf90f4eae.el9.x 3.2 MB/s | 11 MB 00:03 2026-02-22T21:27:34.347 INFO:teuthology.orchestra.run.trial140.stdout:(8/35): python3-cephfs-19.2.3-689.gf90f4eae.el9 2.9 MB/s | 165 kB 00:00 2026-02-22T21:27:34.388 INFO:teuthology.orchestra.run.trial051.stdout:(6/35): python3-ceph-argparse-19.2.3-689.gf90f4 1.0 MB/s | 45 kB 00:00 2026-02-22T21:27:34.398 INFO:teuthology.orchestra.run.trial140.stdout:(9/35): python3-rados-19.2.3-689.gf90f4eae.el9. 6.2 MB/s | 322 kB 00:00 2026-02-22T21:27:34.440 INFO:teuthology.orchestra.run.trial051.stdout:(7/35): python3-ceph-common-19.2.3-689.gf90f4ea 2.7 MB/s | 142 kB 00:00 2026-02-22T21:27:34.450 INFO:teuthology.orchestra.run.trial140.stdout:(10/35): python3-rbd-19.2.3-689.gf90f4eae.el9.x 5.7 MB/s | 303 kB 00:00 2026-02-22T21:27:34.493 INFO:teuthology.orchestra.run.trial051.stdout:(8/35): python3-cephfs-19.2.3-689.gf90f4eae.el9 3.1 MB/s | 165 kB 00:00 2026-02-22T21:27:34.499 INFO:teuthology.orchestra.run.trial140.stdout:(11/35): python3-rgw-19.2.3-689.gf90f4eae.el9.x 2.0 MB/s | 100 kB 00:00 2026-02-22T21:27:34.546 INFO:teuthology.orchestra.run.trial051.stdout:(9/35): python3-rados-19.2.3-689.gf90f4eae.el9. 6.0 MB/s | 322 kB 00:00 2026-02-22T21:27:34.597 INFO:teuthology.orchestra.run.trial051.stdout:(10/35): python3-rbd-19.2.3-689.gf90f4eae.el9.x 5.8 MB/s | 303 kB 00:00 2026-02-22T21:27:34.625 INFO:teuthology.orchestra.run.trial125.stdout:(5/35): librgw2-19.2.3-689.gf90f4eae.el9.x86_64 3.8 MB/s | 5.4 MB 00:01 2026-02-22T21:27:34.643 INFO:teuthology.orchestra.run.trial051.stdout:(11/35): python3-rgw-19.2.3-689.gf90f4eae.el9.x 2.1 MB/s | 100 kB 00:00 2026-02-22T21:27:34.663 INFO:teuthology.orchestra.run.trial051.stdout:(12/35): mailcap-2.1.49-5.el9.noarch.rpm 1.7 MB/s | 33 kB 00:00 2026-02-22T21:27:34.670 INFO:teuthology.orchestra.run.trial125.stdout:(6/35): python3-ceph-argparse-19.2.3-689.gf90f4 1.0 MB/s | 45 kB 00:00 2026-02-22T21:27:34.683 INFO:teuthology.orchestra.run.trial051.stdout:(13/35): nvme-cli-2.16-1.el9.x86_64.rpm 58 MB/s | 1.2 MB 00:00 2026-02-22T21:27:34.688 INFO:teuthology.orchestra.run.trial051.stdout:(14/35): pciutils-3.7.0-7.el9.x86_64.rpm 17 MB/s | 93 kB 00:00 2026-02-22T21:27:34.715 INFO:teuthology.orchestra.run.trial125.stdout:(7/35): python3-ceph-common-19.2.3-689.gf90f4ea 3.1 MB/s | 142 kB 00:00 2026-02-22T21:27:34.753 INFO:teuthology.orchestra.run.trial125.stdout:(8/35): python3-cephfs-19.2.3-689.gf90f4eae.el9 4.2 MB/s | 165 kB 00:00 2026-02-22T21:27:34.797 INFO:teuthology.orchestra.run.trial125.stdout:(9/35): python3-rados-19.2.3-689.gf90f4eae.el9. 7.2 MB/s | 322 kB 00:00 2026-02-22T21:27:34.840 INFO:teuthology.orchestra.run.trial125.stdout:(10/35): python3-rbd-19.2.3-689.gf90f4eae.el9.x 6.9 MB/s | 303 kB 00:00 2026-02-22T21:27:34.861 INFO:teuthology.orchestra.run.trial125.stdout:(11/35): python3-rgw-19.2.3-689.gf90f4eae.el9.x 4.6 MB/s | 100 kB 00:00 2026-02-22T21:27:34.872 INFO:teuthology.orchestra.run.trial125.stdout:(12/35): mailcap-2.1.49-5.el9.noarch.rpm 3.0 MB/s | 33 kB 00:00 2026-02-22T21:27:34.892 INFO:teuthology.orchestra.run.trial125.stdout:(13/35): nvme-cli-2.16-1.el9.x86_64.rpm 57 MB/s | 1.2 MB 00:00 2026-02-22T21:27:34.897 INFO:teuthology.orchestra.run.trial068.stdout:(1/35): ceph-base-19.2.3-689.gf90f4eae.el9.x86_ 2.6 MB/s | 5.5 MB 00:02 2026-02-22T21:27:34.898 INFO:teuthology.orchestra.run.trial125.stdout:(14/35): pciutils-3.7.0-7.el9.x86_64.rpm 18 MB/s | 93 kB 00:00 2026-02-22T21:27:34.914 INFO:teuthology.orchestra.run.trial068.stdout:(2/35): ceph-selinux-19.2.3-689.gf90f4eae.el9.x 1.5 MB/s | 25 kB 00:00 2026-02-22T21:27:35.029 INFO:teuthology.orchestra.run.trial068.stdout:(3/35): libcephfs2-19.2.3-689.gf90f4eae.el9.x86 8.5 MB/s | 999 kB 00:00 2026-02-22T21:27:35.032 INFO:teuthology.orchestra.run.trial051.stdout:(15/35): librgw2-19.2.3-689.gf90f4eae.el9.x86_6 5.9 MB/s | 5.4 MB 00:00 2026-02-22T21:27:35.069 INFO:teuthology.orchestra.run.trial140.stdout:(12/35): mailcap-2.1.49-5.el9.noarch.rpm 58 kB/s | 33 kB 00:00 2026-02-22T21:27:35.075 INFO:teuthology.orchestra.run.trial068.stdout:(4/35): libradosstriper1-19.2.3-689.gf90f4eae.e 11 MB/s | 503 kB 00:00 2026-02-22T21:27:35.076 INFO:teuthology.orchestra.run.trial125.stdout:(15/35): ceph-radosgw-19.2.3-689.gf90f4eae.el9. 2.3 MB/s | 11 MB 00:04 2026-02-22T21:27:35.384 INFO:teuthology.orchestra.run.trial140.stdout:(13/35): librgw2-19.2.3-689.gf90f4eae.el9.x86_6 4.5 MB/s | 5.4 MB 00:01 2026-02-22T21:27:35.430 INFO:teuthology.orchestra.run.trial051.stdout:(16/35): boost-program-options-1.75.0-13.el9.x8 140 kB/s | 104 kB 00:00 2026-02-22T21:27:35.526 INFO:teuthology.orchestra.run.trial068.stdout:(5/35): librgw2-19.2.3-689.gf90f4eae.el9.x86_64 12 MB/s | 5.4 MB 00:00 2026-02-22T21:27:35.539 INFO:teuthology.orchestra.run.trial068.stdout:(6/35): python3-ceph-argparse-19.2.3-689.gf90f4 3.5 MB/s | 45 kB 00:00 2026-02-22T21:27:35.553 INFO:teuthology.orchestra.run.trial068.stdout:(7/35): python3-ceph-common-19.2.3-689.gf90f4ea 10 MB/s | 142 kB 00:00 2026-02-22T21:27:35.563 INFO:teuthology.orchestra.run.trial125.stdout:(16/35): boost-program-options-1.75.0-13.el9.x8 156 kB/s | 104 kB 00:00 2026-02-22T21:27:35.568 INFO:teuthology.orchestra.run.trial068.stdout:(8/35): python3-cephfs-19.2.3-689.gf90f4eae.el9 11 MB/s | 165 kB 00:00 2026-02-22T21:27:35.593 INFO:teuthology.orchestra.run.trial051.stdout:(17/35): libnbd-1.20.3-4.el9.x86_64.rpm 292 kB/s | 164 kB 00:00 2026-02-22T21:27:35.597 INFO:teuthology.orchestra.run.trial068.stdout:(9/35): python3-rados-19.2.3-689.gf90f4eae.el9. 11 MB/s | 322 kB 00:00 2026-02-22T21:27:35.626 INFO:teuthology.orchestra.run.trial068.stdout:(10/35): python3-rbd-19.2.3-689.gf90f4eae.el9.x 10 MB/s | 303 kB 00:00 2026-02-22T21:27:35.640 INFO:teuthology.orchestra.run.trial068.stdout:(11/35): python3-rgw-19.2.3-689.gf90f4eae.el9.x 7.0 MB/s | 100 kB 00:00 2026-02-22T21:27:35.676 INFO:teuthology.orchestra.run.trial051.stdout:(18/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2026-02-22T21:27:35.717 INFO:teuthology.orchestra.run.trial068.stdout:(12/35): mailcap-2.1.49-5.el9.noarch.rpm 431 kB/s | 33 kB 00:00 2026-02-22T21:27:35.797 INFO:teuthology.orchestra.run.trial125.stdout:(17/35): libnbd-1.20.3-4.el9.x86_64.rpm 227 kB/s | 164 kB 00:00 2026-02-22T21:27:35.833 INFO:teuthology.orchestra.run.trial068.stdout:(13/35): nvme-cli-2.16-1.el9.x86_64.rpm 9.9 MB/s | 1.2 MB 00:00 2026-02-22T21:27:35.858 INFO:teuthology.orchestra.run.trial068.stdout:(14/35): pciutils-3.7.0-7.el9.x86_64.rpm 3.8 MB/s | 93 kB 00:00 2026-02-22T21:27:35.904 INFO:teuthology.orchestra.run.trial068.stdout:(15/35): ceph-radosgw-19.2.3-689.gf90f4eae.el9. 3.5 MB/s | 11 MB 00:03 2026-02-22T21:27:35.918 INFO:teuthology.orchestra.run.trial051.stdout:(19/35): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2026-02-22T21:27:35.932 INFO:teuthology.orchestra.run.trial068.stdout:(16/35): boost-program-options-1.75.0-13.el9.x8 1.4 MB/s | 104 kB 00:00 2026-02-22T21:27:35.976 INFO:teuthology.orchestra.run.trial068.stdout:(17/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.6 MB/s | 160 kB 00:00 2026-02-22T21:27:35.985 INFO:teuthology.orchestra.run.trial051.stdout:(20/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 289 kB/s | 160 kB 00:00 2026-02-22T21:27:35.988 INFO:teuthology.orchestra.run.trial068.stdout:(18/35): libnbd-1.20.3-4.el9.x86_64.rpm 1.9 MB/s | 164 kB 00:00 2026-02-22T21:27:35.997 INFO:teuthology.orchestra.run.trial068.stdout:(19/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.0 MB/s | 45 kB 00:00 2026-02-22T21:27:36.002 INFO:teuthology.orchestra.run.trial051.stdout:(21/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2026-02-22T21:27:36.006 INFO:teuthology.orchestra.run.trial140.stdout:(14/35): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-22T21:27:36.021 INFO:teuthology.orchestra.run.trial068.stdout:(20/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-22T21:27:36.041 INFO:teuthology.orchestra.run.trial140.stdout:(15/35): pciutils-3.7.0-7.el9.x86_64.rpm 142 kB/s | 93 kB 00:00 2026-02-22T21:27:36.045 INFO:teuthology.orchestra.run.trial068.stdout:(21/35): qatlib-25.08.0-2.el9.x86_64.rpm 9.9 MB/s | 240 kB 00:00 2026-02-22T21:27:36.060 INFO:teuthology.orchestra.run.trial068.stdout:(22/35): librdkafka-1.6.1-102.el9.x86_64.rpm 9.0 MB/s | 662 kB 00:00 2026-02-22T21:27:36.067 INFO:teuthology.orchestra.run.trial068.stdout:(23/35): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-22T21:27:36.070 INFO:teuthology.orchestra.run.trial140.stdout:(16/35): boost-program-options-1.75.0-13.el9.x8 1.6 MB/s | 104 kB 00:00 2026-02-22T21:27:36.082 INFO:teuthology.orchestra.run.trial068.stdout:(24/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.9 MB/s | 66 kB 00:00 2026-02-22T21:27:36.097 INFO:teuthology.orchestra.run.trial051.stdout:(22/35): ceph-common-19.2.3-689.gf90f4eae.el9.x 4.2 MB/s | 22 MB 00:05 2026-02-22T21:27:36.098 INFO:teuthology.orchestra.run.trial051.stdout:(23/35): qatlib-service-25.08.0-2.el9.x86_64.rp 389 kB/s | 37 kB 00:00 2026-02-22T21:27:36.137 INFO:teuthology.orchestra.run.trial140.stdout:(17/35): ceph-common-19.2.3-689.gf90f4eae.el9.x 3.9 MB/s | 22 MB 00:05 2026-02-22T21:27:36.137 INFO:teuthology.orchestra.run.trial140.stdout:(18/35): libnbd-1.20.3-4.el9.x86_64.rpm 1.7 MB/s | 164 kB 00:00 2026-02-22T21:27:36.137 INFO:teuthology.orchestra.run.trial140.stdout:(19/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2026-02-22T21:27:36.182 INFO:teuthology.orchestra.run.trial051.stdout:(24/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 785 kB/s | 66 kB 00:00 2026-02-22T21:27:36.184 INFO:teuthology.orchestra.run.trial140.stdout:(20/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.2 MB/s | 292 kB 00:00 2026-02-22T21:27:36.191 INFO:teuthology.orchestra.run.trial140.stdout:(21/35): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-22T21:27:36.200 INFO:teuthology.orchestra.run.trial140.stdout:(22/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 710 kB/s | 45 kB 00:00 2026-02-22T21:27:36.208 INFO:teuthology.orchestra.run.trial140.stdout:(23/35): qatlib-25.08.0-2.el9.x86_64.rpm 9.9 MB/s | 240 kB 00:00 2026-02-22T21:27:36.213 INFO:teuthology.orchestra.run.trial140.stdout:(24/35): qatlib-service-25.08.0-2.el9.x86_64.rp 1.6 MB/s | 37 kB 00:00 2026-02-22T21:27:36.223 INFO:teuthology.orchestra.run.trial140.stdout:(25/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.9 MB/s | 66 kB 00:00 2026-02-22T21:27:36.268 INFO:teuthology.orchestra.run.trial140.stdout:(26/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 550 kB/s | 25 kB 00:00 2026-02-22T21:27:36.286 INFO:teuthology.orchestra.run.trial125.stdout:(18/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 222 kB/s | 160 kB 00:00 2026-02-22T21:27:36.288 INFO:teuthology.orchestra.run.trial140.stdout:(27/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.8 MB/s | 308 kB 00:00 2026-02-22T21:27:36.288 INFO:teuthology.orchestra.run.trial140.stdout:(28/35): liboath-2.6.12-1.el9.x86_64.rpm 2.3 MB/s | 49 kB 00:00 2026-02-22T21:27:36.305 INFO:teuthology.orchestra.run.trial140.stdout:(29/35): libunwind-1.6.2-1.el9.x86_64.rpm 4.0 MB/s | 67 kB 00:00 2026-02-22T21:27:36.322 INFO:teuthology.orchestra.run.trial140.stdout:(30/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 24 MB/s | 838 kB 00:00 2026-02-22T21:27:36.323 INFO:teuthology.orchestra.run.trial140.stdout:(31/35): re2-20211101-20.el9.x86_64.rpm 10 MB/s | 191 kB 00:00 2026-02-22T21:27:36.329 INFO:teuthology.orchestra.run.trial140.stdout:(32/35): libarrow-9.0.0-15.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2026-02-22T21:27:36.357 INFO:teuthology.orchestra.run.trial140.stdout:(33/35): thrift-0.15.0-4.el9.x86_64.rpm 46 MB/s | 1.6 MB 00:00 2026-02-22T21:27:36.358 INFO:teuthology.orchestra.run.trial051.stdout:(25/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.2 MB/s | 308 kB 00:00 2026-02-22T21:27:36.379 INFO:teuthology.orchestra.run.trial125.stdout:(19/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 78 kB/s | 45 kB 00:00 2026-02-22T21:27:36.398 INFO:teuthology.orchestra.run.trial051.stdout:(26/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 608 kB/s | 25 kB 00:00 2026-02-22T21:27:36.428 INFO:teuthology.orchestra.run.trial068.stdout:(25/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 852 kB/s | 308 kB 00:00 2026-02-22T21:27:36.449 INFO:teuthology.orchestra.run.trial051.stdout:(27/35): liboath-2.6.12-1.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2026-02-22T21:27:36.482 INFO:teuthology.orchestra.run.trial068.stdout:(26/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 463 kB/s | 25 kB 00:00 2026-02-22T21:27:36.497 INFO:teuthology.orchestra.run.trial051.stdout:(28/35): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2026-02-22T21:27:36.538 INFO:teuthology.orchestra.run.trial068.stdout:(27/35): liboath-2.6.12-1.el9.x86_64.rpm 871 kB/s | 49 kB 00:00 2026-02-22T21:27:36.541 INFO:teuthology.orchestra.run.trial140.stdout:(34/35): librbd1-19.2.3-689.gf90f4eae.el9.x86_6 15 MB/s | 3.2 MB 00:00 2026-02-22T21:27:36.574 INFO:teuthology.orchestra.run.trial125.stdout:(20/35): ceph-common-19.2.3-689.gf90f4eae.el9.x 3.6 MB/s | 22 MB 00:06 2026-02-22T21:27:36.586 INFO:teuthology.orchestra.run.trial051.stdout:(29/35): libarrow-9.0.0-15.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2026-02-22T21:27:36.596 INFO:teuthology.orchestra.run.trial068.stdout:(28/35): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2026-02-22T21:27:36.609 INFO:teuthology.orchestra.run.trial051.stdout:(30/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 7.3 MB/s | 838 kB 00:00 2026-02-22T21:27:36.617 INFO:teuthology.orchestra.run.trial051.stdout:(31/35): qatlib-25.08.0-2.el9.x86_64.rpm 379 kB/s | 240 kB 00:00 2026-02-22T21:27:36.635 INFO:teuthology.orchestra.run.trial068.stdout:(29/35): ceph-common-19.2.3-689.gf90f4eae.el9.x 5.7 MB/s | 22 MB 00:03 2026-02-22T21:27:36.638 INFO:teuthology.orchestra.run.trial140.stdout:(35/35): librados2-19.2.3-689.gf90f4eae.el9.x86 11 MB/s | 3.4 MB 00:00 2026-02-22T21:27:36.639 INFO:teuthology.orchestra.run.trial051.stdout:(32/35): re2-20211101-20.el9.x86_64.rpm 3.5 MB/s | 191 kB 00:00 2026-02-22T21:27:36.641 INFO:teuthology.orchestra.run.trial068.stdout:(30/35): libarrow-9.0.0-15.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2026-02-22T21:27:36.642 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:36.642 INFO:teuthology.orchestra.run.trial140.stdout:Total 9.5 MB/s | 63 MB 00:06 2026-02-22T21:27:36.707 INFO:teuthology.orchestra.run.trial068.stdout:(31/35): thrift-0.15.0-4.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2026-02-22T21:27:36.708 INFO:teuthology.orchestra.run.trial051.stdout:(33/35): thrift-0.15.0-4.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2026-02-22T21:27:36.741 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:27:36.743 INFO:teuthology.orchestra.run.trial068.stdout:(32/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 5.6 MB/s | 838 kB 00:00 2026-02-22T21:27:36.761 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:27:36.761 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:27:36.852 INFO:teuthology.orchestra.run.trial051.stdout:(34/35): librbd1-19.2.3-689.gf90f4eae.el9.x86_6 15 MB/s | 3.2 MB 00:00 2026-02-22T21:27:36.856 INFO:teuthology.orchestra.run.trial051.stdout:(35/35): librados2-19.2.3-689.gf90f4eae.el9.x86 14 MB/s | 3.4 MB 00:00 2026-02-22T21:27:36.859 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:36.859 INFO:teuthology.orchestra.run.trial051.stdout:Total 10 MB/s | 63 MB 00:06 2026-02-22T21:27:36.943 INFO:teuthology.orchestra.run.trial068.stdout:(33/35): re2-20211101-20.el9.x86_64.rpm 621 kB/s | 191 kB 00:00 2026-02-22T21:27:36.949 INFO:teuthology.orchestra.run.trial068.stdout:(34/35): librbd1-19.2.3-689.gf90f4eae.el9.x86_6 15 MB/s | 3.2 MB 00:00 2026-02-22T21:27:36.954 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:27:36.956 INFO:teuthology.orchestra.run.trial068.stdout:(35/35): librados2-19.2.3-689.gf90f4eae.el9.x86 14 MB/s | 3.4 MB 00:00 2026-02-22T21:27:36.957 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:36.958 INFO:teuthology.orchestra.run.trial068.stdout:Total 14 MB/s | 63 MB 00:04 2026-02-22T21:27:36.973 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:27:36.973 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:27:37.014 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:27:37.014 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:27:37.051 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:27:37.071 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:27:37.071 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:27:37.167 INFO:teuthology.orchestra.run.trial125.stdout:(21/35): librdkafka-1.6.1-102.el9.x86_64.rpm 752 kB/s | 662 kB 00:00 2026-02-22T21:27:37.168 INFO:teuthology.orchestra.run.trial125.stdout:(22/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 370 kB/s | 292 kB 00:00 2026-02-22T21:27:37.228 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:27:37.228 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:27:37.306 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:27:37.315 INFO:teuthology.orchestra.run.trial140.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-22T21:27:37.317 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-22T21:27:37.325 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:27:37.326 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:27:37.329 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-22T21:27:37.334 INFO:teuthology.orchestra.run.trial125.stdout:(23/35): qatlib-25.08.0-2.el9.x86_64.rpm 315 kB/s | 240 kB 00:00 2026-02-22T21:27:37.480 INFO:teuthology.orchestra.run.trial140.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-22T21:27:37.485 INFO:teuthology.orchestra.run.trial140.stdout: Upgrading : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.515 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.520 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:27:37.529 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-22T21:27:37.532 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-22T21:27:37.543 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-22T21:27:37.557 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:37.558 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.560 INFO:teuthology.orchestra.run.trial125.stdout:(24/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.3 MB/s | 308 kB 00:00 2026-02-22T21:27:37.570 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.574 INFO:teuthology.orchestra.run.trial140.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-22T21:27:37.583 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 9/37 2026-02-22T21:27:37.605 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:37.606 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.617 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:27:37.619 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.627 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-22T21:27:37.629 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-22T21:27:37.641 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-22T21:27:37.652 INFO:teuthology.orchestra.run.trial140.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-22T21:27:37.656 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-22T21:27:37.657 INFO:teuthology.orchestra.run.trial125.stdout:(25/35): libarrow-9.0.0-15.el9.x86_64.rpm 46 MB/s | 4.4 MB 00:00 2026-02-22T21:27:37.664 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-22T21:27:37.667 INFO:teuthology.orchestra.run.trial140.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-22T21:27:37.670 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-22T21:27:37.676 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-22T21:27:37.679 INFO:teuthology.orchestra.run.trial125.stdout:(26/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.1 MB/s | 25 kB 00:00 2026-02-22T21:27:37.694 INFO:teuthology.orchestra.run.trial051.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-22T21:27:37.699 INFO:teuthology.orchestra.run.trial051.stdout: Upgrading : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.702 INFO:teuthology.orchestra.run.trial125.stdout:(27/35): liboath-2.6.12-1.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2026-02-22T21:27:37.727 INFO:teuthology.orchestra.run.trial125.stdout:(28/35): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2026-02-22T21:27:37.729 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.735 INFO:teuthology.orchestra.run.trial125.stdout:(29/35): qatlib-service-25.08.0-2.el9.x86_64.rp 65 kB/s | 37 kB 00:00 2026-02-22T21:27:37.754 INFO:teuthology.orchestra.run.trial125.stdout:(30/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 31 MB/s | 838 kB 00:00 2026-02-22T21:27:37.759 INFO:teuthology.orchestra.run.trial125.stdout:(31/35): re2-20211101-20.el9.x86_64.rpm 8.1 MB/s | 191 kB 00:00 2026-02-22T21:27:37.772 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:37.773 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.784 INFO:teuthology.orchestra.run.trial125.stdout:(32/35): thrift-0.15.0-4.el9.x86_64.rpm 53 MB/s | 1.6 MB 00:00 2026-02-22T21:27:37.784 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.789 INFO:teuthology.orchestra.run.trial051.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-22T21:27:37.792 INFO:teuthology.orchestra.run.trial068.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-22T21:27:37.793 INFO:teuthology.orchestra.run.trial125.stdout:(33/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 106 kB/s | 66 kB 00:00 2026-02-22T21:27:37.797 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.797 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 9/37 2026-02-22T21:27:37.817 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:37.818 INFO:teuthology.orchestra.run.trial140.stdout: Upgrading : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:37.819 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:37.820 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.824 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:37.832 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.846 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:37.849 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 20/37 2026-02-22T21:27:37.855 INFO:teuthology.orchestra.run.trial140.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-22T21:27:37.863 INFO:teuthology.orchestra.run.trial051.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-22T21:27:37.866 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:37.867 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-22T21:27:37.867 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.873 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-22T21:27:37.875 INFO:teuthology.orchestra.run.trial051.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-22T21:27:37.878 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:37.878 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-22T21:27:37.882 INFO:teuthology.orchestra.run.trial068.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-22T21:27:37.884 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-22T21:27:37.891 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 9/37 2026-02-22T21:27:37.913 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:37.914 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.927 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:37.965 INFO:teuthology.orchestra.run.trial068.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-22T21:27:37.970 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-22T21:27:37.976 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-22T21:27:37.979 INFO:teuthology.orchestra.run.trial068.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-22T21:27:37.981 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-22T21:27:37.986 INFO:teuthology.orchestra.run.trial125.stdout:(34/35): librados2-19.2.3-689.gf90f4eae.el9.x86 15 MB/s | 3.4 MB 00:00 2026-02-22T21:27:37.987 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-22T21:27:38.018 INFO:teuthology.orchestra.run.trial125.stdout:(35/35): librbd1-19.2.3-689.gf90f4eae.el9.x86_6 14 MB/s | 3.2 MB 00:00 2026-02-22T21:27:38.020 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:38.020 INFO:teuthology.orchestra.run.trial125.stdout:Total 8.1 MB/s | 63 MB 00:07 2026-02-22T21:27:38.024 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:38.025 INFO:teuthology.orchestra.run.trial051.stdout: Upgrading : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:38.050 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:38.052 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 20/37 2026-02-22T21:27:38.058 INFO:teuthology.orchestra.run.trial051.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-22T21:27:38.080 INFO:teuthology.orchestra.run.trial140.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-22T21:27:38.082 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.098 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.104 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 24/37 2026-02-22T21:27:38.109 INFO:teuthology.orchestra.run.trial140.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-22T21:27:38.111 INFO:teuthology.orchestra.run.trial140.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-22T21:27:38.112 INFO:teuthology.orchestra.run.trial140.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.117 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:27:38.127 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:38.128 INFO:teuthology.orchestra.run.trial068.stdout: Upgrading : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:38.130 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.135 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:27:38.135 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:27:38.152 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:38.155 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 20/37 2026-02-22T21:27:38.161 INFO:teuthology.orchestra.run.trial068.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-22T21:27:38.249 INFO:teuthology.orchestra.run.trial140.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-22T21:27:38.252 INFO:teuthology.orchestra.run.trial140.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.285 INFO:teuthology.orchestra.run.trial051.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-22T21:27:38.286 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.302 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.306 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 24/37 2026-02-22T21:27:38.310 INFO:teuthology.orchestra.run.trial051.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-22T21:27:38.313 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-22T21:27:38.314 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.332 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.385 INFO:teuthology.orchestra.run.trial068.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-22T21:27:38.386 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.391 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:27:38.391 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:27:38.401 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:38.406 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 24/37 2026-02-22T21:27:38.410 INFO:teuthology.orchestra.run.trial068.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-22T21:27:38.412 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-22T21:27:38.413 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.429 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:38.446 INFO:teuthology.orchestra.run.trial051.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-22T21:27:38.449 INFO:teuthology.orchestra.run.trial051.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.527 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.528 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T21:27:38.528 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:38.542 INFO:teuthology.orchestra.run.trial068.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-22T21:27:38.544 INFO:teuthology.orchestra.run.trial068.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.565 INFO:teuthology.orchestra.run.trial140.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-22T21:27:38.567 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 31/37 2026-02-22T21:27:38.688 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:27:38.697 INFO:teuthology.orchestra.run.trial125.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-22T21:27:38.701 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-22T21:27:38.712 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-22T21:27:38.741 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.741 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T21:27:38.741 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:38.778 INFO:teuthology.orchestra.run.trial051.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-22T21:27:38.780 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 31/37 2026-02-22T21:27:38.832 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:38.832 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T21:27:38.832 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:38.867 INFO:teuthology.orchestra.run.trial068.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-22T21:27:38.869 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 31/37 2026-02-22T21:27:38.871 INFO:teuthology.orchestra.run.trial125.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-22T21:27:38.876 INFO:teuthology.orchestra.run.trial125.stdout: Upgrading : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:38.906 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:38.952 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:38.952 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:38.964 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:38.969 INFO:teuthology.orchestra.run.trial125.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-22T21:27:38.979 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 9/37 2026-02-22T21:27:39.003 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:39.003 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:39.016 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:39.046 INFO:teuthology.orchestra.run.trial125.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-22T21:27:39.050 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-22T21:27:39.058 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-22T21:27:39.060 INFO:teuthology.orchestra.run.trial125.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-22T21:27:39.063 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-22T21:27:39.069 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-22T21:27:39.220 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:39.221 INFO:teuthology.orchestra.run.trial125.stdout: Upgrading : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:39.248 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T21:27:39.250 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 20/37 2026-02-22T21:27:39.256 INFO:teuthology.orchestra.run.trial125.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-22T21:27:39.497 INFO:teuthology.orchestra.run.trial125.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-22T21:27:39.498 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:39.514 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T21:27:39.519 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 24/37 2026-02-22T21:27:39.524 INFO:teuthology.orchestra.run.trial125.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-22T21:27:39.526 INFO:teuthology.orchestra.run.trial125.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-22T21:27:39.527 INFO:teuthology.orchestra.run.trial125.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:39.544 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T21:27:39.552 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.555 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.659 INFO:teuthology.orchestra.run.trial125.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-22T21:27:39.661 INFO:teuthology.orchestra.run.trial125.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:39.761 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.763 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.839 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.842 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.842 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:39.844 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:39.883 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:39.883 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-22T21:27:39.883 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-22T21:27:39.884 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:39.887 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:39.959 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T21:27:39.959 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T21:27:39.960 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:39.996 INFO:teuthology.orchestra.run.trial125.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-22T21:27:39.997 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 31/37 2026-02-22T21:27:40.048 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:40.051 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:40.092 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:40.092 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-22T21:27:40.092 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-22T21:27:40.093 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:40.095 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:40.125 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:40.128 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:40.172 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:40.172 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-22T21:27:40.172 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-22T21:27:40.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:40.174 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:41.039 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:41.041 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:41.340 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T21:27:41.343 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:41.383 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T21:27:41.383 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-22T21:27:41.384 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-22T21:27:41.384 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:41.387 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /sys 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /proc 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /mnt 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /var/tmp 2026-02-22T21:27:44.416 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /home 2026-02-22T21:27:44.417 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /root 2026-02-22T21:27:44.417 INFO:teuthology.orchestra.run.trial140.stdout:skipping the directory /tmp 2026-02-22T21:27:44.417 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:44.585 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:44.585 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /sys 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /proc 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /mnt 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /var/tmp 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /home 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /root 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout:skipping the directory /tmp 2026-02-22T21:27:44.586 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /sys 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /proc 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /mnt 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /var/tmp 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /home 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /root 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout:skipping the directory /tmp 2026-02-22T21:27:44.654 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:44.863 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:44.886 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:44.887 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:27:44.887 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-22T21:27:44.887 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:44.887 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:44.887 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:44.889 INFO:teuthology.orchestra.run.trial140.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:44.901 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:44.901 INFO:teuthology.orchestra.run.trial140.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:44.955 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:44.955 INFO:teuthology.orchestra.run.trial140.stdout:Creating group 'qat' with GID 992. 2026-02-22T21:27:44.955 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:45.036 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:45.059 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.061 INFO:teuthology.orchestra.run.trial051.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:45.073 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:45.073 INFO:teuthology.orchestra.run.trial051.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.101 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:45.125 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.126 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:45.129 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.129 INFO:teuthology.orchestra.run.trial051.stdout:Creating group 'qat' with GID 992. 2026-02-22T21:27:45.129 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.138 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:45.138 INFO:teuthology.orchestra.run.trial068.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.192 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.192 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'qat' with GID 992. 2026-02-22T21:27:45.192 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.239 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 1/37 2026-02-22T21:27:45.239 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2/37 2026-02-22T21:27:45.239 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 3/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 4/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 8/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 9/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 12/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 13/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:45.240 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-22T21:27:45.241 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-22T21:27:45.242 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-22T21:27:45.242 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-22T21:27:45.242 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:45.242 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-22T21:27:45.242 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 36/37 2026-02-22T21:27:45.303 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.303 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:45.303 INFO:teuthology.orchestra.run.trial140.stdout:Upgraded: 2026-02-22T21:27:45.303 INFO:teuthology.orchestra.run.trial140.stdout: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.303 INFO:teuthology.orchestra.run.trial140.stdout: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: re2-1:20211101-20.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:45.304 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:27:45.345 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-test 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 1/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 3/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 4/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 8/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 9/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 12/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 13/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-22T21:27:45.411 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-22T21:27:45.412 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 36/37 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout:Upgraded: 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.472 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-22T21:27:45.473 INFO:teuthology.orchestra.run.trial051.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: re2-1:20211101-20.el9.x86_64 2026-02-22T21:27:45.474 INFO:teuthology.orchestra.run.trial051.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-22T21:27:45.475 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.475 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 1/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 3/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 4/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:45.479 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 8/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 9/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 12/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 13/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-22T21:27:45.480 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-22T21:27:45.481 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 36/37 2026-02-22T21:27:45.514 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-test 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout:Upgraded: 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:27:45.540 INFO:teuthology.orchestra.run.trial068.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-22T21:27:45.541 INFO:teuthology.orchestra.run.trial068.stdout: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.542 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: re2-1:20211101-20.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.543 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:27:45.581 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-test 2026-02-22T21:27:45.653 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:17 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repository Size 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout: ceph-test x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 50 M 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout:Installing dependencies: 2026-02-22T21:27:45.736 INFO:teuthology.orchestra.run.trial140.stdout: libcephsqlite x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 163 k 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:Install 5 Packages 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 50 M 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 212 M 2026-02-22T21:27:45.737 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:27:45.824 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:16 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /sys 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /proc 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /mnt 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /var/tmp 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /home 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /root 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout:skipping the directory /tmp 2026-02-22T21:27:45.846 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:45.885 INFO:teuthology.orchestra.run.trial140.stdout:(1/5): libcephsqlite-19.2.3-689.gf90f4eae.el9.x 1.8 MB/s | 163 kB 00:00 2026-02-22T21:27:45.896 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:14 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:27:45.906 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 50 M 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: libcephsqlite x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 163 k 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout:Install 5 Packages 2026-02-22T21:27:45.907 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:45.908 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 50 M 2026-02-22T21:27:45.908 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 212 M 2026-02-22T21:27:45.908 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:27:45.911 INFO:teuthology.orchestra.run.trial140.stdout:(2/5): libxslt-1.1.34-12.el9.x86_64.rpm 2.0 MB/s | 233 kB 00:00 2026-02-22T21:27:45.933 INFO:teuthology.orchestra.run.trial140.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.8 MB/s | 64 kB 00:00 2026-02-22T21:27:45.979 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:27:45.980 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:45.980 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-22T21:27:45.980 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:45.980 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:27:45.980 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 50 M 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 163 k 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Install 5 Packages 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 50 M 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 212 M 2026-02-22T21:27:45.981 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:27:46.007 INFO:teuthology.orchestra.run.trial140.stdout:(4/5): socat-1.7.4.1-8.el9.x86_64.rpm 2.4 MB/s | 303 kB 00:00 2026-02-22T21:27:46.188 INFO:teuthology.orchestra.run.trial068.stdout:(1/5): libxslt-1.1.34-12.el9.x86_64.rpm 1.9 MB/s | 233 kB 00:00 2026-02-22T21:27:46.233 INFO:teuthology.orchestra.run.trial068.stdout:(2/5): socat-1.7.4.1-8.el9.x86_64.rpm 6.7 MB/s | 303 kB 00:00 2026-02-22T21:27:46.254 INFO:teuthology.orchestra.run.trial068.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.9 MB/s | 64 kB 00:00 2026-02-22T21:27:46.319 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:27:46.343 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:46.344 INFO:teuthology.orchestra.run.trial125.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:46.356 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T21:27:46.357 INFO:teuthology.orchestra.run.trial125.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:46.364 INFO:teuthology.orchestra.run.trial051.stdout:(1/5): libcephsqlite-19.2.3-689.gf90f4eae.el9.x 416 kB/s | 163 kB 00:00 2026-02-22T21:27:46.409 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:46.409 INFO:teuthology.orchestra.run.trial125.stdout:Creating group 'qat' with GID 992. 2026-02-22T21:27:46.409 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:46.460 INFO:teuthology.orchestra.run.trial051.stdout:(2/5): libxslt-1.1.34-12.el9.x86_64.rpm 476 kB/s | 233 kB 00:00 2026-02-22T21:27:46.497 INFO:teuthology.orchestra.run.trial068.stdout:(4/5): libcephsqlite-19.2.3-689.gf90f4eae.el9.x 382 kB/s | 163 kB 00:00 2026-02-22T21:27:46.552 INFO:teuthology.orchestra.run.trial051.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 690 kB/s | 64 kB 00:00 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 1/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 3/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 4/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 8/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 9/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 12/37 2026-02-22T21:27:46.696 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 13/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-22T21:27:46.697 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-22T21:27:46.698 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 36/37 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout:Upgraded: 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.759 INFO:teuthology.orchestra.run.trial125.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: re2-1:20211101-20.el9.x86_64 2026-02-22T21:27:46.760 INFO:teuthology.orchestra.run.trial125.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-22T21:27:46.761 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:46.761 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:27:46.800 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-test 2026-02-22T21:27:46.863 INFO:teuthology.orchestra.run.trial140.stdout:(5/5): ceph-test-19.2.3-689.gf90f4eae.el9.x86_6 47 MB/s | 50 MB 00:01 2026-02-22T21:27:46.864 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:46.865 INFO:teuthology.orchestra.run.trial140.stdout:Total 45 MB/s | 50 MB 00:01 2026-02-22T21:27:46.886 INFO:teuthology.orchestra.run.trial051.stdout:(4/5): socat-1.7.4.1-8.el9.x86_64.rpm 581 kB/s | 303 kB 00:00 2026-02-22T21:27:46.891 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:27:46.901 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:27:46.901 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:27:47.089 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:27:47.089 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:27:47.107 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:18 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repository Size 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:27:47.189 INFO:teuthology.orchestra.run.trial125.stdout: ceph-test x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 50 M 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Installing dependencies: 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: libcephsqlite x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 163 k 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Install 5 Packages 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 50 M 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 212 M 2026-02-22T21:27:47.190 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:27:47.301 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:27:47.307 INFO:teuthology.orchestra.run.trial140.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-22T21:27:47.311 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-22T21:27:47.319 INFO:teuthology.orchestra.run.trial140.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-22T21:27:47.320 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:47.366 INFO:teuthology.orchestra.run.trial125.stdout:(1/5): libxslt-1.1.34-12.el9.x86_64.rpm 2.9 MB/s | 233 kB 00:00 2026-02-22T21:27:47.395 INFO:teuthology.orchestra.run.trial125.stdout:(2/5): socat-1.7.4.1-8.el9.x86_64.rpm 10 MB/s | 303 kB 00:00 2026-02-22T21:27:47.411 INFO:teuthology.orchestra.run.trial125.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 4.0 MB/s | 64 kB 00:00 2026-02-22T21:27:47.502 INFO:teuthology.orchestra.run.trial125.stdout:(4/5): libcephsqlite-19.2.3-689.gf90f4eae.el9.x 761 kB/s | 163 kB 00:00 2026-02-22T21:27:48.642 INFO:teuthology.orchestra.run.trial051.stdout:(5/5): ceph-test-19.2.3-689.gf90f4eae.el9.x86_6 19 MB/s | 50 MB 00:02 2026-02-22T21:27:48.644 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:48.644 INFO:teuthology.orchestra.run.trial051.stdout:Total 18 MB/s | 50 MB 00:02 2026-02-22T21:27:48.669 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:27:48.678 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:27:48.678 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:27:48.788 INFO:teuthology.orchestra.run.trial068.stdout:(5/5): ceph-test-19.2.3-689.gf90f4eae.el9.x86_6 18 MB/s | 50 MB 00:02 2026-02-22T21:27:48.789 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:48.790 INFO:teuthology.orchestra.run.trial068.stdout:Total 18 MB/s | 50 MB 00:02 2026-02-22T21:27:48.816 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:27:48.825 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:27:48.825 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:27:48.863 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:27:48.863 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:27:49.009 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:27:49.009 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:27:49.073 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:27:49.079 INFO:teuthology.orchestra.run.trial051.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-22T21:27:49.083 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-22T21:27:49.091 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-22T21:27:49.092 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:49.220 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:27:49.226 INFO:teuthology.orchestra.run.trial068.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-22T21:27:49.230 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-22T21:27:49.238 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-22T21:27:49.239 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:49.522 INFO:teuthology.orchestra.run.trial125.stdout:(5/5): ceph-test-19.2.3-689.gf90f4eae.el9.x86_6 22 MB/s | 50 MB 00:02 2026-02-22T21:27:49.523 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:49.523 INFO:teuthology.orchestra.run.trial125.stdout:Total 22 MB/s | 50 MB 00:02 2026-02-22T21:27:49.550 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:27:49.559 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:27:49.559 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:27:49.567 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:49.585 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:49.746 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:27:49.747 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:27:49.817 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:49.817 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 1/5 2026-02-22T21:27:49.818 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2/5 2026-02-22T21:27:49.818 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-22T21:27:49.818 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-22T21:27:49.873 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-22T21:27:49.873 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:49.874 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:27:49.906 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph 2026-02-22T21:27:49.958 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:27:49.964 INFO:teuthology.orchestra.run.trial125.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-22T21:27:49.968 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-22T21:27:49.976 INFO:teuthology.orchestra.run.trial125.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-22T21:27:49.977 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:50.207 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:22 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:27:50.293 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout:====================================================================================== 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repository Size 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout:====================================================================================== 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout: ceph x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.5 k 2026-02-22T21:27:50.296 INFO:teuthology.orchestra.run.trial140.stdout:Installing dependencies: 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-grafana-dashboards noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 31 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mds x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 2.4 M 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-modules-core noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 253 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mon x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 4.7 M 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-osd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 17 M 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: ceph-prometheus-alerts noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 17 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 769 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-22T21:27:50.297 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-22T21:27:50.298 INFO:teuthology.orchestra.run.trial140.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-22T21:27:50.299 INFO:teuthology.orchestra.run.trial140.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-22T21:27:50.300 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-22T21:27:50.301 INFO:teuthology.orchestra.run.trial140.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-22T21:27:50.302 INFO:teuthology.orchestra.run.trial140.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-22T21:27:50.303 INFO:teuthology.orchestra.run.trial140.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout:Installing weak dependencies: 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 150 k 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-dashboard noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 3.8 M 2026-02-22T21:27:50.304 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 7.4 M 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 22 k 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-rook noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 49 k 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: ceph-volume noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 299 k 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout:====================================================================================== 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout:Install 99 Packages 2026-02-22T21:27:50.305 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:27:50.306 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 95 M 2026-02-22T21:27:50.306 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 520 M 2026-02-22T21:27:50.306 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:27:50.900 INFO:teuthology.orchestra.run.trial140.stdout:(1/99): ceph-19.2.3-689.gf90f4eae.el9.x86_64.rp 125 kB/s | 6.5 kB 00:00 2026-02-22T21:27:50.991 INFO:teuthology.orchestra.run.trial140.stdout:(2/99): ceph-mgr-19.2.3-689.gf90f4eae.el9.x86_6 7.6 MB/s | 1.1 MB 00:00 2026-02-22T21:27:51.017 INFO:teuthology.orchestra.run.trial140.stdout:(3/99): ceph-mds-19.2.3-689.gf90f4eae.el9.x86_6 14 MB/s | 2.4 MB 00:00 2026-02-22T21:27:51.050 INFO:teuthology.orchestra.run.trial140.stdout:(4/99): ceph-grafana-dashboards-19.2.3-689.gf90 955 kB/s | 31 kB 00:00 2026-02-22T21:27:51.087 INFO:teuthology.orchestra.run.trial140.stdout:(5/99): ceph-mgr-cephadm-19.2.3-689.gf90f4eae.e 3.9 MB/s | 150 kB 00:00 2026-02-22T21:27:51.127 INFO:teuthology.orchestra.run.trial140.stdout:(6/99): ceph-mon-19.2.3-689.gf90f4eae.el9.x86_6 21 MB/s | 4.7 MB 00:00 2026-02-22T21:27:51.287 INFO:teuthology.orchestra.run.trial140.stdout:(7/99): ceph-mgr-dashboard-19.2.3-689.gf90f4eae 19 MB/s | 3.8 MB 00:00 2026-02-22T21:27:51.320 INFO:teuthology.orchestra.run.trial140.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-689.gf90f4eae 675 kB/s | 22 kB 00:00 2026-02-22T21:27:51.343 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:51.361 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:51.366 INFO:teuthology.orchestra.run.trial140.stdout:(9/99): ceph-mgr-modules-core-19.2.3-689.gf90f4 5.4 MB/s | 253 kB 00:00 2026-02-22T21:27:51.405 INFO:teuthology.orchestra.run.trial140.stdout:(10/99): ceph-mgr-rook-19.2.3-689.gf90f4eae.el9 1.3 MB/s | 49 kB 00:00 2026-02-22T21:27:51.422 INFO:teuthology.orchestra.run.trial140.stdout:(11/99): ceph-mgr-diskprediction-local-19.2.3-6 25 MB/s | 7.4 MB 00:00 2026-02-22T21:27:51.425 INFO:teuthology.orchestra.run.trial140.stdout:(12/99): ceph-prometheus-alerts-19.2.3-689.gf90 801 kB/s | 17 kB 00:00 2026-02-22T21:27:51.445 INFO:teuthology.orchestra.run.trial140.stdout:(13/99): ceph-volume-19.2.3-689.gf90f4eae.el9.n 13 MB/s | 299 kB 00:00 2026-02-22T21:27:51.462 INFO:teuthology.orchestra.run.trial140.stdout:(14/99): cephadm-19.2.3-689.gf90f4eae.el9.noarc 21 MB/s | 769 kB 00:00 2026-02-22T21:27:51.531 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:51.549 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:51.582 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:51.582 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 1/5 2026-02-22T21:27:51.583 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2/5 2026-02-22T21:27:51.583 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-22T21:27:51.583 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-22T21:27:51.603 INFO:teuthology.orchestra.run.trial140.stdout:(15/99): ceph-osd-19.2.3-689.gf90f4eae.el9.x86_ 28 MB/s | 17 MB 00:00 2026-02-22T21:27:51.636 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-22T21:27:51.636 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:51.636 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:51.637 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:27:51.668 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph 2026-02-22T21:27:51.761 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:51.761 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 1/5 2026-02-22T21:27:51.761 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2/5 2026-02-22T21:27:51.761 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-22T21:27:51.761 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-22T21:27:51.814 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-22T21:27:51.815 INFO:teuthology.orchestra.run.trial068.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-22T21:27:51.816 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:51.816 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:27:51.846 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph 2026-02-22T21:27:51.974 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:22 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:27:52.025 INFO:teuthology.orchestra.run.trial140.stdout:(16/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 72 kB/s | 40 kB 00:00 2026-02-22T21:27:52.060 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout:====================================================================================== 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repository Size 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout:====================================================================================== 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: ceph x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.5 k 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: ceph-grafana-dashboards noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 31 k 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 2.4 M 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:27:52.063 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-modules-core noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 253 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 4.7 M 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 17 M 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: ceph-prometheus-alerts noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 17 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 769 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-22T21:27:52.064 INFO:teuthology.orchestra.run.trial051.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-22T21:27:52.065 INFO:teuthology.orchestra.run.trial051.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-22T21:27:52.066 INFO:teuthology.orchestra.run.trial051.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-22T21:27:52.067 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-22T21:27:52.068 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout:Installing weak dependencies: 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 150 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-dashboard noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 3.8 M 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 7.4 M 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 22 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-rook noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 49 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: ceph-volume noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 299 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout:====================================================================================== 2026-02-22T21:27:52.069 INFO:teuthology.orchestra.run.trial051.stdout:Install 99 Packages 2026-02-22T21:27:52.070 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:27:52.070 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 95 M 2026-02-22T21:27:52.070 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 520 M 2026-02-22T21:27:52.070 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:27:52.165 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:21 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:27:52.245 INFO:teuthology.orchestra.run.trial140.stdout:(17/99): libconfig-1.7.2-9.el9.x86_64.rpm 112 kB/s | 72 kB 00:00 2026-02-22T21:27:52.252 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:27:52.254 INFO:teuthology.orchestra.run.trial068.stdout:====================================================================================== 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repository Size 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout:====================================================================================== 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: ceph x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.5 k 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 31 k 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 2.4 M 2026-02-22T21:27:52.255 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 253 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 4.7 M 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 17 M 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 17 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 769 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-22T21:27:52.256 INFO:teuthology.orchestra.run.trial068.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-22T21:27:52.257 INFO:teuthology.orchestra.run.trial068.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-22T21:27:52.258 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-22T21:27:52.259 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-22T21:27:52.260 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-22T21:27:52.261 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-22T21:27:52.262 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-22T21:27:52.263 INFO:teuthology.orchestra.run.trial068.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 150 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 3.8 M 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 7.4 M 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 22 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 49 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 299 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-22T21:27:52.264 INFO:teuthology.orchestra.run.trial068.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-22T21:27:52.265 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:52.265 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:27:52.265 INFO:teuthology.orchestra.run.trial068.stdout:====================================================================================== 2026-02-22T21:27:52.265 INFO:teuthology.orchestra.run.trial068.stdout:Install 99 Packages 2026-02-22T21:27:52.265 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:27:52.266 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 95 M 2026-02-22T21:27:52.266 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 520 M 2026-02-22T21:27:52.266 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:27:52.273 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T21:27:52.292 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:52.422 INFO:teuthology.orchestra.run.trial140.stdout:(18/99): libgfortran-11.5.0-14.el9.x86_64.rpm 2.0 MB/s | 794 kB 00:00 2026-02-22T21:27:52.493 INFO:teuthology.orchestra.run.trial140.stdout:(19/99): libquadmath-11.5.0-14.el9.x86_64.rpm 744 kB/s | 184 kB 00:00 2026-02-22T21:27:52.523 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T21:27:52.523 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 1/5 2026-02-22T21:27:52.524 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2/5 2026-02-22T21:27:52.524 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-22T21:27:52.524 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-22T21:27:52.576 INFO:teuthology.orchestra.run.trial125.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-22T21:27:52.577 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:52.577 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:27:52.581 INFO:teuthology.orchestra.run.trial140.stdout:(20/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2026-02-22T21:27:52.610 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph 2026-02-22T21:27:52.634 INFO:teuthology.orchestra.run.trial068.stdout:(1/99): ceph-19.2.3-689.gf90f4eae.el9.x86_64.rp 127 kB/s | 6.5 kB 00:00 2026-02-22T21:27:52.669 INFO:teuthology.orchestra.run.trial051.stdout:(1/99): ceph-19.2.3-689.gf90f4eae.el9.x86_64.rp 127 kB/s | 6.5 kB 00:00 2026-02-22T21:27:52.731 INFO:teuthology.orchestra.run.trial068.stdout:(2/99): ceph-mgr-19.2.3-689.gf90f4eae.el9.x86_6 7.3 MB/s | 1.1 MB 00:00 2026-02-22T21:27:52.741 INFO:teuthology.orchestra.run.trial140.stdout:(21/99): python3-ply-3.11-14.el9.noarch.rpm 670 kB/s | 106 kB 00:00 2026-02-22T21:27:52.761 INFO:teuthology.orchestra.run.trial068.stdout:(3/99): ceph-mds-19.2.3-689.gf90f4eae.el9.x86_6 14 MB/s | 2.4 MB 00:00 2026-02-22T21:27:52.801 INFO:teuthology.orchestra.run.trial068.stdout:(4/99): ceph-grafana-dashboards-19.2.3-689.gf90 782 kB/s | 31 kB 00:00 2026-02-22T21:27:52.832 INFO:teuthology.orchestra.run.trial140.stdout:(22/99): python3-cryptography-36.0.1-5.el9.x86_ 3.7 MB/s | 1.2 MB 00:00 2026-02-22T21:27:52.845 INFO:teuthology.orchestra.run.trial068.stdout:(5/99): ceph-mgr-cephadm-19.2.3-689.gf90f4eae.e 3.3 MB/s | 150 kB 00:00 2026-02-22T21:27:52.900 INFO:teuthology.orchestra.run.trial140.stdout:(23/99): python3-pycparser-2.20-6.el9.noarch.rp 847 kB/s | 135 kB 00:00 2026-02-22T21:27:52.913 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:23 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:27:52.943 INFO:teuthology.orchestra.run.trial068.stdout:(6/99): ceph-mon-19.2.3-689.gf90f4eae.el9.x86_6 15 MB/s | 4.7 MB 00:00 2026-02-22T21:27:52.993 INFO:teuthology.orchestra.run.trial140.stdout:(24/99): python3-pyparsing-2.4.7-9.el9.noarch.r 939 kB/s | 150 kB 00:00 2026-02-22T21:27:52.998 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:27:53.001 INFO:teuthology.orchestra.run.trial125.stdout:====================================================================================== 2026-02-22T21:27:53.001 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repository Size 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout:====================================================================================== 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.5 k 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout:Installing dependencies: 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph-grafana-dashboards noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 31 k 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mds x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 2.4 M 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-modules-core noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 253 k 2026-02-22T21:27:53.002 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mon x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 4.7 M 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: ceph-osd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 17 M 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: ceph-prometheus-alerts noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 17 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 769 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-22T21:27:53.003 INFO:teuthology.orchestra.run.trial125.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-22T21:27:53.004 INFO:teuthology.orchestra.run.trial125.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-22T21:27:53.005 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-22T21:27:53.006 INFO:teuthology.orchestra.run.trial125.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-22T21:27:53.007 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-22T21:27:53.008 INFO:teuthology.orchestra.run.trial125.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout:Installing weak dependencies: 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 150 k 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-dashboard noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 3.8 M 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 7.4 M 2026-02-22T21:27:53.009 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 22 k 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-rook noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 49 k 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: ceph-volume noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 299 k 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout:====================================================================================== 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout:Install 99 Packages 2026-02-22T21:27:53.010 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:27:53.011 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 95 M 2026-02-22T21:27:53.011 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 520 M 2026-02-22T21:27:53.011 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:27:53.013 INFO:teuthology.orchestra.run.trial051.stdout:(2/99): ceph-mgr-19.2.3-689.gf90f4eae.el9.x86_6 2.7 MB/s | 1.1 MB 00:00 2026-02-22T21:27:53.058 INFO:teuthology.orchestra.run.trial140.stdout:(25/99): python3-requests-2.25.1-10.el9.noarch. 799 kB/s | 126 kB 00:00 2026-02-22T21:27:53.085 INFO:teuthology.orchestra.run.trial140.stdout:(26/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 214 kB/s | 351 kB 00:01 2026-02-22T21:27:53.146 INFO:teuthology.orchestra.run.trial068.stdout:(7/99): ceph-mgr-dashboard-19.2.3-689.gf90f4eae 13 MB/s | 3.8 MB 00:00 2026-02-22T21:27:53.152 INFO:teuthology.orchestra.run.trial140.stdout:(27/99): python3-urllib3-1.26.5-7.el9.noarch.rp 1.3 MB/s | 218 kB 00:00 2026-02-22T21:27:53.201 INFO:teuthology.orchestra.run.trial068.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-689.gf90f4eae 400 kB/s | 22 kB 00:00 2026-02-22T21:27:53.245 INFO:teuthology.orchestra.run.trial140.stdout:(28/99): zip-3.0-35.el9.x86_64.rpm 1.6 MB/s | 266 kB 00:00 2026-02-22T21:27:53.264 INFO:teuthology.orchestra.run.trial068.stdout:(9/99): ceph-mgr-modules-core-19.2.3-689.gf90f4 3.9 MB/s | 253 kB 00:00 2026-02-22T21:27:53.276 INFO:teuthology.orchestra.run.trial140.stdout:(29/99): flexiblas-3.0.4-9.el9.x86_64.rpm 240 kB/s | 30 kB 00:00 2026-02-22T21:27:53.313 INFO:teuthology.orchestra.run.trial051.stdout:(3/99): ceph-mds-19.2.3-689.gf90f4eae.el9.x86_6 3.5 MB/s | 2.4 MB 00:00 2026-02-22T21:27:53.325 INFO:teuthology.orchestra.run.trial068.stdout:(10/99): ceph-mgr-rook-19.2.3-689.gf90f4eae.el9 810 kB/s | 49 kB 00:00 2026-02-22T21:27:53.354 INFO:teuthology.orchestra.run.trial140.stdout:(30/99): flexiblas-openblas-openmp-3.0.4-9.el9. 189 kB/s | 15 kB 00:00 2026-02-22T21:27:53.367 INFO:teuthology.orchestra.run.trial051.stdout:(4/99): ceph-grafana-dashboards-19.2.3-689.gf90 588 kB/s | 31 kB 00:00 2026-02-22T21:27:53.372 INFO:teuthology.orchestra.run.trial068.stdout:(11/99): ceph-mgr-diskprediction-local-19.2.3-6 17 MB/s | 7.4 MB 00:00 2026-02-22T21:27:53.377 INFO:teuthology.orchestra.run.trial068.stdout:(12/99): ceph-prometheus-alerts-19.2.3-689.gf90 320 kB/s | 17 kB 00:00 2026-02-22T21:27:53.409 INFO:teuthology.orchestra.run.trial051.stdout:(5/99): ceph-mgr-cephadm-19.2.3-689.gf90f4eae.e 3.5 MB/s | 150 kB 00:00 2026-02-22T21:27:53.418 INFO:teuthology.orchestra.run.trial068.stdout:(13/99): ceph-volume-19.2.3-689.gf90f4eae.el9.n 6.2 MB/s | 299 kB 00:00 2026-02-22T21:27:53.441 INFO:teuthology.orchestra.run.trial068.stdout:(14/99): ceph-osd-19.2.3-689.gf90f4eae.el9.x86_ 24 MB/s | 17 MB 00:00 2026-02-22T21:27:53.443 INFO:teuthology.orchestra.run.trial068.stdout:(15/99): cephadm-19.2.3-689.gf90f4eae.el9.noarc 11 MB/s | 769 kB 00:00 2026-02-22T21:27:53.470 INFO:teuthology.orchestra.run.trial068.stdout:(16/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.4 MB/s | 40 kB 00:00 2026-02-22T21:27:53.478 INFO:teuthology.orchestra.run.trial068.stdout:(17/99): libconfig-1.7.2-9.el9.x86_64.rpm 2.0 MB/s | 72 kB 00:00 2026-02-22T21:27:53.485 INFO:teuthology.orchestra.run.trial068.stdout:(18/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 5.2 MB/s | 351 kB 00:00 2026-02-22T21:27:53.496 INFO:teuthology.orchestra.run.trial068.stdout:(19/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 21 MB/s | 253 kB 00:00 2026-02-22T21:27:53.497 INFO:teuthology.orchestra.run.trial068.stdout:(20/99): libquadmath-11.5.0-14.el9.x86_64.rpm 9.7 MB/s | 184 kB 00:00 2026-02-22T21:27:53.503 INFO:teuthology.orchestra.run.trial140.stdout:(31/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 1.6 MB/s | 246 kB 00:00 2026-02-22T21:27:53.508 INFO:teuthology.orchestra.run.trial068.stdout:(21/99): libgfortran-11.5.0-14.el9.x86_64.rpm 20 MB/s | 794 kB 00:00 2026-02-22T21:27:53.508 INFO:teuthology.orchestra.run.trial068.stdout:(22/99): python3-ply-3.11-14.el9.noarch.rpm 9.1 MB/s | 106 kB 00:00 2026-02-22T21:27:53.519 INFO:teuthology.orchestra.run.trial068.stdout:(23/99): python3-pycparser-2.20-6.el9.noarch.rp 12 MB/s | 135 kB 00:00 2026-02-22T21:27:53.520 INFO:teuthology.orchestra.run.trial068.stdout:(24/99): python3-pyparsing-2.4.7-9.el9.noarch.r 13 MB/s | 150 kB 00:00 2026-02-22T21:27:53.529 INFO:teuthology.orchestra.run.trial068.stdout:(25/99): python3-requests-2.25.1-10.el9.noarch. 13 MB/s | 126 kB 00:00 2026-02-22T21:27:53.531 INFO:teuthology.orchestra.run.trial068.stdout:(26/99): python3-cryptography-36.0.1-5.el9.x86_ 35 MB/s | 1.2 MB 00:00 2026-02-22T21:27:53.533 INFO:teuthology.orchestra.run.trial068.stdout:(27/99): python3-urllib3-1.26.5-7.el9.noarch.rp 16 MB/s | 218 kB 00:00 2026-02-22T21:27:53.540 INFO:teuthology.orchestra.run.trial068.stdout:(28/99): unzip-6.0-59.el9.x86_64.rpm 16 MB/s | 182 kB 00:00 2026-02-22T21:27:53.543 INFO:teuthology.orchestra.run.trial068.stdout:(29/99): zip-3.0-35.el9.x86_64.rpm 23 MB/s | 266 kB 00:00 2026-02-22T21:27:53.604 INFO:teuthology.orchestra.run.trial140.stdout:(32/99): unzip-6.0-59.el9.x86_64.rpm 333 kB/s | 182 kB 00:00 2026-02-22T21:27:53.607 INFO:teuthology.orchestra.run.trial051.stdout:(6/99): ceph-mgr-dashboard-19.2.3-689.gf90f4eae 19 MB/s | 3.8 MB 00:00 2026-02-22T21:27:53.617 INFO:teuthology.orchestra.run.trial140.stdout:(33/99): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2026-02-22T21:27:53.652 INFO:teuthology.orchestra.run.trial125.stdout:(1/99): ceph-19.2.3-689.gf90f4eae.el9.x86_64.rp 74 kB/s | 6.5 kB 00:00 2026-02-22T21:27:53.674 INFO:teuthology.orchestra.run.trial051.stdout:(7/99): ceph-mon-19.2.3-689.gf90f4eae.el9.x86_6 4.7 MB/s | 4.7 MB 00:01 2026-02-22T21:27:53.692 INFO:teuthology.orchestra.run.trial051.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-689.gf90f4eae 1.2 MB/s | 22 kB 00:00 2026-02-22T21:27:53.719 INFO:teuthology.orchestra.run.trial051.stdout:(9/99): ceph-mgr-modules-core-19.2.3-689.gf90f4 9.3 MB/s | 253 kB 00:00 2026-02-22T21:27:53.730 INFO:teuthology.orchestra.run.trial140.stdout:(34/99): openblas-0.3.29-1.el9.x86_64.rpm 332 kB/s | 42 kB 00:00 2026-02-22T21:27:53.745 INFO:teuthology.orchestra.run.trial051.stdout:(10/99): ceph-mgr-rook-19.2.3-689.gf90f4eae.el9 1.8 MB/s | 49 kB 00:00 2026-02-22T21:27:53.753 INFO:teuthology.orchestra.run.trial140.stdout:(35/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.9 MB/s | 3.0 MB 00:00 2026-02-22T21:27:53.773 INFO:teuthology.orchestra.run.trial051.stdout:(11/99): ceph-prometheus-alerts-19.2.3-689.gf90 594 kB/s | 17 kB 00:00 2026-02-22T21:27:53.809 INFO:teuthology.orchestra.run.trial051.stdout:(12/99): ceph-volume-19.2.3-689.gf90f4eae.el9.n 8.3 MB/s | 299 kB 00:00 2026-02-22T21:27:53.920 INFO:teuthology.orchestra.run.trial051.stdout:(13/99): cephadm-19.2.3-689.gf90f4eae.el9.noarc 6.8 MB/s | 769 kB 00:00 2026-02-22T21:27:53.937 INFO:teuthology.orchestra.run.trial125.stdout:(2/99): ceph-mgr-19.2.3-689.gf90f4eae.el9.x86_6 2.9 MB/s | 1.1 MB 00:00 2026-02-22T21:27:53.982 INFO:teuthology.orchestra.run.trial140.stdout:(36/99): protobuf-3.14.0-17.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2026-02-22T21:27:53.985 INFO:teuthology.orchestra.run.trial051.stdout:(14/99): ceph-mgr-diskprediction-local-19.2.3-6 20 MB/s | 7.4 MB 00:00 2026-02-22T21:27:54.023 INFO:teuthology.orchestra.run.trial068.stdout:(30/99): flexiblas-openblas-openmp-3.0.4-9.el9. 31 kB/s | 15 kB 00:00 2026-02-22T21:27:54.071 INFO:teuthology.orchestra.run.trial125.stdout:(3/99): ceph-mds-19.2.3-689.gf90f4eae.el9.x86_6 4.8 MB/s | 2.4 MB 00:00 2026-02-22T21:27:54.080 INFO:teuthology.orchestra.run.trial051.stdout:(15/99): ceph-osd-19.2.3-689.gf90f4eae.el9.x86_ 16 MB/s | 17 MB 00:01 2026-02-22T21:27:54.082 INFO:teuthology.orchestra.run.trial125.stdout:(4/99): ceph-grafana-dashboards-19.2.3-689.gf90 2.6 MB/s | 31 kB 00:00 2026-02-22T21:27:54.086 INFO:teuthology.orchestra.run.trial068.stdout:(31/99): flexiblas-3.0.4-9.el9.x86_64.rpm 54 kB/s | 30 kB 00:00 2026-02-22T21:27:54.094 INFO:teuthology.orchestra.run.trial125.stdout:(5/99): ceph-mgr-cephadm-19.2.3-689.gf90f4eae.e 12 MB/s | 150 kB 00:00 2026-02-22T21:27:54.138 INFO:teuthology.orchestra.run.trial140.stdout:(37/99): python3-devel-3.9.25-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2026-02-22T21:27:54.223 INFO:teuthology.orchestra.run.trial140.stdout:(38/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 8.7 MB/s | 5.3 MB 00:00 2026-02-22T21:27:54.269 INFO:teuthology.orchestra.run.trial125.stdout:(6/99): ceph-mgr-dashboard-19.2.3-689.gf90f4eae 22 MB/s | 3.8 MB 00:00 2026-02-22T21:27:54.300 INFO:teuthology.orchestra.run.trial125.stdout:(7/99): ceph-mon-19.2.3-689.gf90f4eae.el9.x86_6 7.3 MB/s | 4.7 MB 00:00 2026-02-22T21:27:54.317 INFO:teuthology.orchestra.run.trial125.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-689.gf90f4eae 1.3 MB/s | 22 kB 00:00 2026-02-22T21:27:54.325 INFO:teuthology.orchestra.run.trial140.stdout:(39/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 1.3 MB/s | 249 kB 00:00 2026-02-22T21:27:54.340 INFO:teuthology.orchestra.run.trial125.stdout:(9/99): ceph-mgr-modules-core-19.2.3-689.gf90f4 11 MB/s | 253 kB 00:00 2026-02-22T21:27:54.360 INFO:teuthology.orchestra.run.trial125.stdout:(10/99): ceph-mgr-rook-19.2.3-689.gf90f4eae.el9 2.4 MB/s | 49 kB 00:00 2026-02-22T21:27:54.361 INFO:teuthology.orchestra.run.trial140.stdout:(40/99): python3-libstoragemgmt-1.10.1-1.el9.x8 1.3 MB/s | 177 kB 00:00 2026-02-22T21:27:54.380 INFO:teuthology.orchestra.run.trial125.stdout:(11/99): ceph-prometheus-alerts-19.2.3-689.gf90 864 kB/s | 17 kB 00:00 2026-02-22T21:27:54.406 INFO:teuthology.orchestra.run.trial125.stdout:(12/99): ceph-volume-19.2.3-689.gf90f4eae.el9.n 11 MB/s | 299 kB 00:00 2026-02-22T21:27:54.438 INFO:teuthology.orchestra.run.trial051.stdout:(16/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 89 kB/s | 40 kB 00:00 2026-02-22T21:27:54.475 INFO:teuthology.orchestra.run.trial140.stdout:(41/99): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 6.0 MB 00:00 2026-02-22T21:27:54.486 INFO:teuthology.orchestra.run.trial125.stdout:(13/99): cephadm-19.2.3-689.gf90f4eae.el9.noarc 9.4 MB/s | 769 kB 00:00 2026-02-22T21:27:54.513 INFO:teuthology.orchestra.run.trial140.stdout:(42/99): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2026-02-22T21:27:54.566 INFO:teuthology.orchestra.run.trial051.stdout:(17/99): libconfig-1.7.2-9.el9.x86_64.rpm 149 kB/s | 72 kB 00:00 2026-02-22T21:27:54.572 INFO:teuthology.orchestra.run.trial125.stdout:(14/99): ceph-mgr-diskprediction-local-19.2.3-6 24 MB/s | 7.4 MB 00:00 2026-02-22T21:27:54.587 INFO:teuthology.orchestra.run.trial068.stdout:(32/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.9 MB/s | 3.0 MB 00:01 2026-02-22T21:27:54.613 INFO:teuthology.orchestra.run.trial140.stdout:(43/99): python3-markupsafe-1.1.1-12.el9.x86_64 251 kB/s | 35 kB 00:00 2026-02-22T21:27:54.624 INFO:teuthology.orchestra.run.trial140.stdout:(44/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 4.1 MB/s | 1.2 MB 00:00 2026-02-22T21:27:54.648 INFO:teuthology.orchestra.run.trial125.stdout:(15/99): ceph-osd-19.2.3-689.gf90f4eae.el9.x86_ 24 MB/s | 17 MB 00:00 2026-02-22T21:27:54.648 INFO:teuthology.orchestra.run.trial125.stdout:(16/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 2.1 MB/s | 351 kB 00:00 2026-02-22T21:27:54.649 INFO:teuthology.orchestra.run.trial125.stdout:(17/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 526 kB/s | 40 kB 00:00 2026-02-22T21:27:54.669 INFO:teuthology.orchestra.run.trial125.stdout:(18/99): libconfig-1.7.2-9.el9.x86_64.rpm 3.3 MB/s | 72 kB 00:00 2026-02-22T21:27:54.692 INFO:teuthology.orchestra.run.trial125.stdout:(19/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 11 MB/s | 253 kB 00:00 2026-02-22T21:27:54.727 INFO:teuthology.orchestra.run.trial140.stdout:(45/99): python3-packaging-20.9-5.el9.noarch.rp 753 kB/s | 77 kB 00:00 2026-02-22T21:27:54.728 INFO:teuthology.orchestra.run.trial051.stdout:(18/99): libquadmath-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 184 kB 00:00 2026-02-22T21:27:54.742 INFO:teuthology.orchestra.run.trial125.stdout:(20/99): libquadmath-11.5.0-14.el9.x86_64.rpm 1.9 MB/s | 184 kB 00:00 2026-02-22T21:27:54.755 INFO:teuthology.orchestra.run.trial125.stdout:(21/99): python3-cryptography-36.0.1-5.el9.x86_ 20 MB/s | 1.2 MB 00:00 2026-02-22T21:27:54.757 INFO:teuthology.orchestra.run.trial125.stdout:(22/99): libgfortran-11.5.0-14.el9.x86_64.rpm 7.1 MB/s | 794 kB 00:00 2026-02-22T21:27:54.763 INFO:teuthology.orchestra.run.trial125.stdout:(23/99): python3-ply-3.11-14.el9.noarch.rpm 4.8 MB/s | 106 kB 00:00 2026-02-22T21:27:54.779 INFO:teuthology.orchestra.run.trial125.stdout:(24/99): python3-pycparser-2.20-6.el9.noarch.rp 5.7 MB/s | 135 kB 00:00 2026-02-22T21:27:54.782 INFO:teuthology.orchestra.run.trial125.stdout:(25/99): python3-pyparsing-2.4.7-9.el9.noarch.r 6.0 MB/s | 150 kB 00:00 2026-02-22T21:27:54.785 INFO:teuthology.orchestra.run.trial125.stdout:(26/99): python3-requests-2.25.1-10.el9.noarch. 5.8 MB/s | 126 kB 00:00 2026-02-22T21:27:54.786 INFO:teuthology.orchestra.run.trial068.stdout:(33/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 324 kB/s | 246 kB 00:00 2026-02-22T21:27:54.802 INFO:teuthology.orchestra.run.trial125.stdout:(27/99): python3-urllib3-1.26.5-7.el9.noarch.rp 9.2 MB/s | 218 kB 00:00 2026-02-22T21:27:54.806 INFO:teuthology.orchestra.run.trial125.stdout:(28/99): unzip-6.0-59.el9.x86_64.rpm 7.4 MB/s | 182 kB 00:00 2026-02-22T21:27:54.812 INFO:teuthology.orchestra.run.trial125.stdout:(29/99): flexiblas-3.0.4-9.el9.x86_64.rpm 2.8 MB/s | 30 kB 00:00 2026-02-22T21:27:54.814 INFO:teuthology.orchestra.run.trial051.stdout:(19/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 253 kB 00:00 2026-02-22T21:27:54.816 INFO:teuthology.orchestra.run.trial125.stdout:(30/99): flexiblas-openblas-openmp-3.0.4-9.el9. 3.7 MB/s | 15 kB 00:00 2026-02-22T21:27:54.821 INFO:teuthology.orchestra.run.trial068.stdout:(34/99): lua-5.4.4-4.el9.x86_64.rpm 256 kB/s | 188 kB 00:00 2026-02-22T21:27:54.824 INFO:teuthology.orchestra.run.trial140.stdout:(46/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 2.0 MB/s | 442 kB 00:00 2026-02-22T21:27:54.824 INFO:teuthology.orchestra.run.trial125.stdout:(31/99): zip-3.0-35.el9.x86_64.rpm 6.6 MB/s | 266 kB 00:00 2026-02-22T21:27:54.829 INFO:teuthology.orchestra.run.trial125.stdout:(32/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 20 MB/s | 246 kB 00:00 2026-02-22T21:27:54.833 INFO:teuthology.orchestra.run.trial125.stdout:(33/99): openblas-0.3.29-1.el9.x86_64.rpm 9.4 MB/s | 42 kB 00:00 2026-02-22T21:27:54.842 INFO:teuthology.orchestra.run.trial125.stdout:(34/99): lua-5.4.4-4.el9.x86_64.rpm 10 MB/s | 188 kB 00:00 2026-02-22T21:27:54.848 INFO:teuthology.orchestra.run.trial125.stdout:(35/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 71 MB/s | 3.0 MB 00:00 2026-02-22T21:27:54.871 INFO:teuthology.orchestra.run.trial125.stdout:(36/99): protobuf-3.14.0-17.el9.x86_64.rpm 34 MB/s | 1.0 MB 00:00 2026-02-22T21:27:54.879 INFO:teuthology.orchestra.run.trial125.stdout:(37/99): python3-devel-3.9.25-3.el9.x86_64.rpm 34 MB/s | 244 kB 00:00 2026-02-22T21:27:54.883 INFO:teuthology.orchestra.run.trial140.stdout:(47/99): python3-protobuf-3.14.0-17.el9.noarch. 1.7 MB/s | 267 kB 00:00 2026-02-22T21:27:54.887 INFO:teuthology.orchestra.run.trial125.stdout:(38/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 27 MB/s | 249 kB 00:00 2026-02-22T21:27:54.895 INFO:teuthology.orchestra.run.trial125.stdout:(39/99): python3-babel-2.9.1-2.el9.noarch.rpm 128 MB/s | 6.0 MB 00:00 2026-02-22T21:27:54.896 INFO:teuthology.orchestra.run.trial125.stdout:(40/99): python3-libstoragemgmt-1.10.1-1.el9.x8 20 MB/s | 177 kB 00:00 2026-02-22T21:27:54.904 INFO:teuthology.orchestra.run.trial125.stdout:(41/99): python3-mako-1.1.4-6.el9.noarch.rpm 22 MB/s | 172 kB 00:00 2026-02-22T21:27:54.908 INFO:teuthology.orchestra.run.trial125.stdout:(42/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 95 MB/s | 1.2 MB 00:00 2026-02-22T21:27:54.909 INFO:teuthology.orchestra.run.trial125.stdout:(43/99): python3-markupsafe-1.1.1-12.el9.x86_64 7.1 MB/s | 35 kB 00:00 2026-02-22T21:27:54.917 INFO:teuthology.orchestra.run.trial125.stdout:(44/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 50 MB/s | 442 kB 00:00 2026-02-22T21:27:54.923 INFO:teuthology.orchestra.run.trial125.stdout:(45/99): python3-packaging-20.9-5.el9.noarch.rp 14 MB/s | 77 kB 00:00 2026-02-22T21:27:54.929 INFO:teuthology.orchestra.run.trial125.stdout:(46/99): python3-protobuf-3.14.0-17.el9.noarch. 43 MB/s | 267 kB 00:00 2026-02-22T21:27:54.942 INFO:teuthology.orchestra.run.trial125.stdout:(47/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 181 MB/s | 6.1 MB 00:00 2026-02-22T21:27:54.943 INFO:teuthology.orchestra.run.trial125.stdout:(48/99): python3-pyasn1-0.4.8-6.el9.noarch.rpm 12 MB/s | 159 kB 00:00 2026-02-22T21:27:54.948 INFO:teuthology.orchestra.run.trial125.stdout:(49/99): python3-requests-oauthlib-1.3.0-12.el9 10 MB/s | 54 kB 00:00 2026-02-22T21:27:54.950 INFO:teuthology.orchestra.run.trial125.stdout:(50/99): python3-pyasn1-modules-0.4.8-6.el9.noa 35 MB/s | 279 kB 00:00 2026-02-22T21:27:54.955 INFO:teuthology.orchestra.run.trial125.stdout:(51/99): python3-toml-0.10.2-6.el9.noarch.rpm 8.4 MB/s | 42 kB 00:00 2026-02-22T21:27:54.960 INFO:teuthology.orchestra.run.trial125.stdout:(52/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 35 MB/s | 189 kB 00:00 2026-02-22T21:27:54.971 INFO:teuthology.orchestra.run.trial125.stdout:(53/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 38 MB/s | 5.3 MB 00:00 2026-02-22T21:27:54.971 INFO:teuthology.orchestra.run.trial125.stdout:(54/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 8.2 MB/s | 90 kB 00:00 2026-02-22T21:27:54.980 INFO:teuthology.orchestra.run.trial051.stdout:(20/99): python3-cryptography-36.0.1-5.el9.x86_ 7.5 MB/s | 1.2 MB 00:00 2026-02-22T21:27:55.000 INFO:teuthology.orchestra.run.trial140.stdout:(48/99): python3-pyasn1-0.4.8-6.el9.noarch.rpm 905 kB/s | 159 kB 00:00 2026-02-22T21:27:55.050 INFO:teuthology.orchestra.run.trial140.stdout:(49/99): python3-pyasn1-modules-0.4.8-6.el9.noa 1.6 MB/s | 279 kB 00:00 2026-02-22T21:27:55.057 INFO:teuthology.orchestra.run.trial125.stdout:(55/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 177 MB/s | 19 MB 00:00 2026-02-22T21:27:55.065 INFO:teuthology.orchestra.run.trial051.stdout:(21/99): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2026-02-22T21:27:55.150 INFO:teuthology.orchestra.run.trial051.stdout:(22/99): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 135 kB 00:00 2026-02-22T21:27:55.155 INFO:teuthology.orchestra.run.trial140.stdout:(50/99): python3-requests-oauthlib-1.3.0-12.el9 344 kB/s | 54 kB 00:00 2026-02-22T21:27:55.165 INFO:teuthology.orchestra.run.trial140.stdout:(51/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 9.4 MB/s | 6.1 MB 00:00 2026-02-22T21:27:55.167 INFO:teuthology.orchestra.run.trial068.stdout:(35/99): openblas-0.3.29-1.el9.x86_64.rpm 73 kB/s | 42 kB 00:00 2026-02-22T21:27:55.235 INFO:teuthology.orchestra.run.trial051.stdout:(23/99): python3-pyparsing-2.4.7-9.el9.noarch.r 1.7 MB/s | 150 kB 00:00 2026-02-22T21:27:55.254 INFO:teuthology.orchestra.run.trial140.stdout:(52/99): python3-toml-0.10.2-6.el9.noarch.rpm 424 kB/s | 42 kB 00:00 2026-02-22T21:27:55.293 INFO:teuthology.orchestra.run.trial140.stdout:(53/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.4 MB/s | 189 kB 00:00 2026-02-22T21:27:55.319 INFO:teuthology.orchestra.run.trial051.stdout:(24/99): python3-requests-2.25.1-10.el9.noarch. 1.5 MB/s | 126 kB 00:00 2026-02-22T21:27:55.336 INFO:teuthology.orchestra.run.trial051.stdout:(25/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 248 kB/s | 351 kB 00:01 2026-02-22T21:27:55.360 INFO:teuthology.orchestra.run.trial140.stdout:(54/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 849 kB/s | 90 kB 00:00 2026-02-22T21:27:55.374 INFO:teuthology.orchestra.run.trial125.stdout:(56/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.7 MB/s | 551 kB 00:00 2026-02-22T21:27:55.404 INFO:teuthology.orchestra.run.trial051.stdout:(26/99): python3-urllib3-1.26.5-7.el9.noarch.rp 2.5 MB/s | 218 kB 00:00 2026-02-22T21:27:55.412 INFO:teuthology.orchestra.run.trial125.stdout:(57/99): grpc-data-1.46.7-10.el9.noarch.rpm 511 kB/s | 19 kB 00:00 2026-02-22T21:27:55.456 INFO:teuthology.orchestra.run.trial125.stdout:(58/99): luarocks-3.9.2-5.el9.noarch.rpm 3.4 MB/s | 151 kB 00:00 2026-02-22T21:27:55.489 INFO:teuthology.orchestra.run.trial051.stdout:(27/99): zip-3.0-35.el9.x86_64.rpm 3.1 MB/s | 266 kB 00:00 2026-02-22T21:27:55.519 INFO:teuthology.orchestra.run.trial125.stdout:(59/99): python3-asyncssh-2.13.2-5.el9.noarch.r 8.5 MB/s | 548 kB 00:00 2026-02-22T21:27:55.527 INFO:teuthology.orchestra.run.trial140.stdout:(55/99): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2026-02-22T21:27:55.529 INFO:teuthology.orchestra.run.trial125.stdout:(60/99): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2026-02-22T21:27:55.558 INFO:teuthology.orchestra.run.trial125.stdout:(61/99): python3-autocommand-2.2.2-8.el9.noarch 776 kB/s | 29 kB 00:00 2026-02-22T21:27:55.569 INFO:teuthology.orchestra.run.trial051.stdout:(28/99): flexiblas-3.0.4-9.el9.x86_64.rpm 368 kB/s | 30 kB 00:00 2026-02-22T21:27:55.596 INFO:teuthology.orchestra.run.trial125.stdout:(62/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.1 MB/s | 43 kB 00:00 2026-02-22T21:27:55.634 INFO:teuthology.orchestra.run.trial125.stdout:(63/99): python3-cachetools-4.2.4-1.el9.noarch. 844 kB/s | 32 kB 00:00 2026-02-22T21:27:55.672 INFO:teuthology.orchestra.run.trial125.stdout:(64/99): python3-certifi-2023.05.07-4.el9.noarc 374 kB/s | 14 kB 00:00 2026-02-22T21:27:55.716 INFO:teuthology.orchestra.run.trial125.stdout:(65/99): python3-cheroot-10.0.1-4.el9.noarch.rp 3.9 MB/s | 173 kB 00:00 2026-02-22T21:27:55.725 INFO:teuthology.orchestra.run.trial125.stdout:(66/99): python3-backports-tarfile-1.2.0-1.el9. 309 kB/s | 60 kB 00:00 2026-02-22T21:27:55.742 INFO:teuthology.orchestra.run.trial068.stdout:(36/99): protobuf-3.14.0-17.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2026-02-22T21:27:55.757 INFO:teuthology.orchestra.run.trial051.stdout:(29/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 16 MB/s | 3.0 MB 00:00 2026-02-22T21:27:55.768 INFO:teuthology.orchestra.run.trial125.stdout:(67/99): python3-cherrypy-18.6.1-2.el9.noarch.r 6.8 MB/s | 358 kB 00:00 2026-02-22T21:27:55.782 INFO:teuthology.orchestra.run.trial051.stdout:(30/99): flexiblas-openblas-openmp-3.0.4-9.el9. 589 kB/s | 15 kB 00:00 2026-02-22T21:27:55.809 INFO:teuthology.orchestra.run.trial051.stdout:(31/99): unzip-6.0-59.el9.x86_64.rpm 385 kB/s | 182 kB 00:00 2026-02-22T21:27:55.810 INFO:teuthology.orchestra.run.trial051.stdout:(32/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 8.8 MB/s | 246 kB 00:00 2026-02-22T21:27:55.835 INFO:teuthology.orchestra.run.trial051.stdout:(33/99): openblas-0.3.29-1.el9.x86_64.rpm 1.6 MB/s | 42 kB 00:00 2026-02-22T21:27:55.845 INFO:teuthology.orchestra.run.trial125.stdout:(68/99): python3-google-auth-2.45.0-1.el9.noarc 2.1 MB/s | 254 kB 00:00 2026-02-22T21:27:55.885 INFO:teuthology.orchestra.run.trial125.stdout:(69/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 17 MB/s | 2.0 MB 00:00 2026-02-22T21:27:55.890 INFO:teuthology.orchestra.run.trial125.stdout:(70/99): protobuf-compiler-3.14.0-17.el9.x86_64 939 kB/s | 862 kB 00:00 2026-02-22T21:27:55.894 INFO:teuthology.orchestra.run.trial125.stdout:(71/99): python3-grpcio-tools-1.46.7-10.el9.x86 2.8 MB/s | 144 kB 00:00 2026-02-22T21:27:55.899 INFO:teuthology.orchestra.run.trial068.stdout:(37/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 4.7 MB/s | 5.3 MB 00:01 2026-02-22T21:27:55.901 INFO:teuthology.orchestra.run.trial051.stdout:(34/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 80 MB/s | 5.3 MB 00:00 2026-02-22T21:27:55.912 INFO:teuthology.orchestra.run.trial140.stdout:(56/99): protobuf-compiler-3.14.0-17.el9.x86_64 1.5 MB/s | 862 kB 00:00 2026-02-22T21:27:55.924 INFO:teuthology.orchestra.run.trial125.stdout:(72/99): python3-influxdb-5.3.1-1.el9.noarch.rp 3.5 MB/s | 139 kB 00:00 2026-02-22T21:27:55.930 INFO:teuthology.orchestra.run.trial051.stdout:(35/99): protobuf-3.14.0-17.el9.x86_64.rpm 34 MB/s | 1.0 MB 00:00 2026-02-22T21:27:55.932 INFO:teuthology.orchestra.run.trial125.stdout:(73/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 281 kB/s | 11 kB 00:00 2026-02-22T21:27:55.955 INFO:teuthology.orchestra.run.trial051.stdout:(36/99): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2026-02-22T21:27:55.962 INFO:teuthology.orchestra.run.trial125.stdout:(74/99): python3-jaraco-classes-3.2.1-5.el9.noa 468 kB/s | 18 kB 00:00 2026-02-22T21:27:55.971 INFO:teuthology.orchestra.run.trial125.stdout:(75/99): python3-jaraco-collections-3.0.0-8.el9 595 kB/s | 23 kB 00:00 2026-02-22T21:27:55.996 INFO:teuthology.orchestra.run.trial051.stdout:(37/99): python3-babel-2.9.1-2.el9.noarch.rpm 91 MB/s | 6.0 MB 00:00 2026-02-22T21:27:55.999 INFO:teuthology.orchestra.run.trial125.stdout:(76/99): python3-jaraco-context-6.0.1-3.el9.noa 521 kB/s | 20 kB 00:00 2026-02-22T21:27:56.002 INFO:teuthology.orchestra.run.trial140.stdout:(57/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.1 MB/s | 551 kB 00:00 2026-02-22T21:27:56.007 INFO:teuthology.orchestra.run.trial051.stdout:(38/99): python3-devel-3.9.25-3.el9.x86_64.rpm 4.5 MB/s | 244 kB 00:00 2026-02-22T21:27:56.010 INFO:teuthology.orchestra.run.trial125.stdout:(77/99): python3-jaraco-functools-3.5.0-2.el9.n 510 kB/s | 19 kB 00:00 2026-02-22T21:27:56.024 INFO:teuthology.orchestra.run.trial051.stdout:(39/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 8.6 MB/s | 249 kB 00:00 2026-02-22T21:27:56.037 INFO:teuthology.orchestra.run.trial125.stdout:(78/99): python3-jaraco-text-4.0.0-2.el9.noarch 699 kB/s | 26 kB 00:00 2026-02-22T21:27:56.039 INFO:teuthology.orchestra.run.trial051.stdout:(40/99): python3-libstoragemgmt-1.10.1-1.el9.x8 5.5 MB/s | 177 kB 00:00 2026-02-22T21:27:56.055 INFO:teuthology.orchestra.run.trial051.stdout:(41/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 40 MB/s | 1.2 MB 00:00 2026-02-22T21:27:56.074 INFO:teuthology.orchestra.run.trial140.stdout:(58/99): luarocks-3.9.2-5.el9.noarch.rpm 2.1 MB/s | 151 kB 00:00 2026-02-22T21:27:56.075 INFO:teuthology.orchestra.run.trial125.stdout:(79/99): python3-logutils-0.3.5-21.el9.noarch.r 1.2 MB/s | 46 kB 00:00 2026-02-22T21:27:56.081 INFO:teuthology.orchestra.run.trial051.stdout:(42/99): python3-markupsafe-1.1.1-12.el9.x86_64 1.3 MB/s | 35 kB 00:00 2026-02-22T21:27:56.085 INFO:teuthology.orchestra.run.trial125.stdout:(80/99): python3-isodate-0.6.1-3.el9.noarch.rpm 287 kB/s | 56 kB 00:00 2026-02-22T21:27:56.085 INFO:teuthology.orchestra.run.trial051.stdout:(43/99): python3-mako-1.1.4-6.el9.noarch.rpm 3.6 MB/s | 172 kB 00:00 2026-02-22T21:27:56.096 INFO:teuthology.orchestra.run.trial140.stdout:(59/99): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2026-02-22T21:27:56.113 INFO:teuthology.orchestra.run.trial125.stdout:(81/99): python3-more-itertools-8.12.0-2.el9.no 2.0 MB/s | 79 kB 00:00 2026-02-22T21:27:56.128 INFO:teuthology.orchestra.run.trial051.stdout:(44/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 10 MB/s | 442 kB 00:00 2026-02-22T21:27:56.129 INFO:teuthology.orchestra.run.trial125.stdout:(82/99): python3-kubernetes-26.1.0-3.el9.noarch 8.6 MB/s | 1.0 MB 00:00 2026-02-22T21:27:56.146 INFO:teuthology.orchestra.run.trial051.stdout:(45/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 95 MB/s | 6.1 MB 00:00 2026-02-22T21:27:56.151 INFO:teuthology.orchestra.run.trial125.stdout:(83/99): python3-natsort-7.1.1-5.el9.noarch.rpm 1.5 MB/s | 58 kB 00:00 2026-02-22T21:27:56.155 INFO:teuthology.orchestra.run.trial051.stdout:(46/99): python3-packaging-20.9-5.el9.noarch.rp 2.8 MB/s | 77 kB 00:00 2026-02-22T21:27:56.162 INFO:teuthology.orchestra.run.trial125.stdout:(84/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2026-02-22T21:27:56.172 INFO:teuthology.orchestra.run.trial125.stdout:(85/99): python3-pecan-1.4.2-3.el9.noarch.rpm 6.0 MB/s | 272 kB 00:00 2026-02-22T21:27:56.174 INFO:teuthology.orchestra.run.trial051.stdout:(47/99): python3-protobuf-3.14.0-17.el9.noarch. 9.3 MB/s | 267 kB 00:00 2026-02-22T21:27:56.178 INFO:teuthology.orchestra.run.trial140.stdout:(60/99): python3-asyncssh-2.13.2-5.el9.noarch.r 5.2 MB/s | 548 kB 00:00 2026-02-22T21:27:56.184 INFO:teuthology.orchestra.run.trial051.stdout:(48/99): python3-pyasn1-0.4.8-6.el9.noarch.rpm 5.3 MB/s | 159 kB 00:00 2026-02-22T21:27:56.189 INFO:teuthology.orchestra.run.trial125.stdout:(86/99): python3-portend-3.1.0-2.el9.noarch.rpm 435 kB/s | 16 kB 00:00 2026-02-22T21:27:56.202 INFO:teuthology.orchestra.run.trial051.stdout:(49/99): python3-pyasn1-modules-0.4.8-6.el9.noa 9.7 MB/s | 279 kB 00:00 2026-02-22T21:27:56.210 INFO:teuthology.orchestra.run.trial125.stdout:(87/99): python3-repoze-lru-0.7-16.el9.noarch.r 814 kB/s | 31 kB 00:00 2026-02-22T21:27:56.210 INFO:teuthology.orchestra.run.trial051.stdout:(50/99): python3-requests-oauthlib-1.3.0-12.el9 2.0 MB/s | 54 kB 00:00 2026-02-22T21:27:56.211 INFO:teuthology.orchestra.run.trial140.stdout:(61/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 17 MB/s | 19 MB 00:01 2026-02-22T21:27:56.211 INFO:teuthology.orchestra.run.trial140.stdout:(62/99): python3-autocommand-2.2.2-8.el9.noarch 255 kB/s | 29 kB 00:00 2026-02-22T21:27:56.214 INFO:teuthology.orchestra.run.trial125.stdout:(88/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2026-02-22T21:27:56.234 INFO:teuthology.orchestra.run.trial125.stdout:(89/99): python3-routes-2.5.1-5.el9.noarch.rpm 4.1 MB/s | 188 kB 00:00 2026-02-22T21:27:56.236 INFO:teuthology.orchestra.run.trial051.stdout:(51/99): python3-toml-0.10.2-6.el9.noarch.rpm 1.6 MB/s | 42 kB 00:00 2026-02-22T21:27:56.242 INFO:teuthology.orchestra.run.trial140.stdout:(63/99): python3-backports-tarfile-1.2.0-1.el9. 930 kB/s | 60 kB 00:00 2026-02-22T21:27:56.249 INFO:teuthology.orchestra.run.trial125.stdout:(90/99): python3-rsa-4.9-2.el9.noarch.rpm 1.5 MB/s | 59 kB 00:00 2026-02-22T21:27:56.266 INFO:teuthology.orchestra.run.trial051.stdout:(52/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 6.1 MB/s | 189 kB 00:00 2026-02-22T21:27:56.271 INFO:teuthology.orchestra.run.trial125.stdout:(91/99): python3-tempora-5.0.0-2.el9.noarch.rpm 949 kB/s | 36 kB 00:00 2026-02-22T21:27:56.272 INFO:teuthology.orchestra.run.trial125.stdout:(92/99): python3-saml-1.16.0-1.el9.noarch.rpm 2.1 MB/s | 125 kB 00:00 2026-02-22T21:27:56.288 INFO:teuthology.orchestra.run.trial125.stdout:(93/99): python3-typing-extensions-4.15.0-1.el9 2.1 MB/s | 86 kB 00:00 2026-02-22T21:27:56.293 INFO:teuthology.orchestra.run.trial051.stdout:(53/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.3 MB/s | 90 kB 00:00 2026-02-22T21:27:56.300 INFO:teuthology.orchestra.run.trial068.stdout:(38/99): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2026-02-22T21:27:56.304 INFO:teuthology.orchestra.run.trial140.stdout:(64/99): python3-certifi-2023.05.07-4.el9.noarc 228 kB/s | 14 kB 00:00 2026-02-22T21:27:56.312 INFO:teuthology.orchestra.run.trial125.stdout:(94/99): python3-webob-1.8.8-2.el9.noarch.rpm 5.6 MB/s | 230 kB 00:00 2026-02-22T21:27:56.319 INFO:teuthology.orchestra.run.trial051.stdout:(54/99): lua-devel-5.4.4-4.el9.x86_64.rpm 889 kB/s | 22 kB 00:00 2026-02-22T21:27:56.321 INFO:teuthology.orchestra.run.trial140.stdout:(65/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 394 kB/s | 43 kB 00:00 2026-02-22T21:27:56.323 INFO:teuthology.orchestra.run.trial125.stdout:(95/99): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2026-02-22T21:27:56.336 INFO:teuthology.orchestra.run.trial125.stdout:(96/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.8 MB/s | 427 kB 00:00 2026-02-22T21:27:56.350 INFO:teuthology.orchestra.run.trial125.stdout:(97/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.2 MB/s | 48 kB 00:00 2026-02-22T21:27:56.363 INFO:teuthology.orchestra.run.trial125.stdout:(98/99): python3-xmltodict-0.12.0-15.el9.noarch 549 kB/s | 22 kB 00:00 2026-02-22T21:27:56.372 INFO:teuthology.orchestra.run.trial051.stdout:(55/99): protobuf-compiler-3.14.0-17.el9.x86_64 16 MB/s | 862 kB 00:00 2026-02-22T21:27:56.374 INFO:teuthology.orchestra.run.trial125.stdout:(99/99): python3-zc-lockfile-2.0-10.el9.noarch. 522 kB/s | 20 kB 00:00 2026-02-22T21:27:56.376 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:56.376 INFO:teuthology.orchestra.run.trial125.stdout:Total 28 MB/s | 95 MB 00:03 2026-02-22T21:27:56.377 INFO:teuthology.orchestra.run.trial140.stdout:(66/99): python3-cheroot-10.0.1-4.el9.noarch.rp 2.3 MB/s | 173 kB 00:00 2026-02-22T21:27:56.398 INFO:teuthology.orchestra.run.trial051.stdout:(56/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 98 MB/s | 19 MB 00:00 2026-02-22T21:27:56.399 INFO:teuthology.orchestra.run.trial051.stdout:(57/99): libgfortran-11.5.0-14.el9.x86_64.rpm 405 kB/s | 794 kB 00:01 2026-02-22T21:27:56.417 INFO:teuthology.orchestra.run.trial140.stdout:(67/99): python3-cachetools-4.2.4-1.el9.noarch. 156 kB/s | 32 kB 00:00 2026-02-22T21:27:56.454 INFO:teuthology.orchestra.run.trial140.stdout:(68/99): python3-google-auth-2.45.0-1.el9.noarc 3.2 MB/s | 254 kB 00:00 2026-02-22T21:27:56.510 INFO:teuthology.orchestra.run.trial068.stdout:(39/99): python3-devel-3.9.25-3.el9.x86_64.rpm 318 kB/s | 244 kB 00:00 2026-02-22T21:27:56.525 INFO:teuthology.orchestra.run.trial140.stdout:(69/99): python3-grpcio-tools-1.46.7-10.el9.x86 2.0 MB/s | 144 kB 00:00 2026-02-22T21:27:56.569 INFO:teuthology.orchestra.run.trial140.stdout:(70/99): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2026-02-22T21:27:56.595 INFO:teuthology.orchestra.run.trial140.stdout:(71/99): python3-influxdb-5.3.1-1.el9.noarch.rp 1.9 MB/s | 139 kB 00:00 2026-02-22T21:27:56.607 INFO:teuthology.orchestra.run.trial051.stdout:(58/99): grpc-data-1.46.7-10.el9.noarch.rpm 93 kB/s | 19 kB 00:00 2026-02-22T21:27:56.635 INFO:teuthology.orchestra.run.trial140.stdout:(72/99): python3-isodate-0.6.1-3.el9.noarch.rpm 844 kB/s | 56 kB 00:00 2026-02-22T21:27:56.657 INFO:teuthology.orchestra.run.trial140.stdout:(73/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 173 kB/s | 11 kB 00:00 2026-02-22T21:27:56.664 INFO:teuthology.orchestra.run.trial068.stdout:(40/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 325 kB/s | 249 kB 00:00 2026-02-22T21:27:56.698 INFO:teuthology.orchestra.run.trial140.stdout:(74/99): python3-jaraco-classes-3.2.1-5.el9.noa 282 kB/s | 18 kB 00:00 2026-02-22T21:27:56.718 INFO:teuthology.orchestra.run.trial140.stdout:(75/99): python3-jaraco-collections-3.0.0-8.el9 374 kB/s | 23 kB 00:00 2026-02-22T21:27:56.757 INFO:teuthology.orchestra.run.trial051.stdout:(59/99): luarocks-3.9.2-5.el9.noarch.rpm 423 kB/s | 151 kB 00:00 2026-02-22T21:27:56.761 INFO:teuthology.orchestra.run.trial140.stdout:(76/99): python3-jaraco-context-6.0.1-3.el9.noa 312 kB/s | 20 kB 00:00 2026-02-22T21:27:56.780 INFO:teuthology.orchestra.run.trial140.stdout:(77/99): python3-jaraco-functools-3.5.0-2.el9.n 315 kB/s | 19 kB 00:00 2026-02-22T21:27:56.793 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:27:56.825 INFO:teuthology.orchestra.run.trial140.stdout:(78/99): python3-jaraco-text-4.0.0-2.el9.noarch 415 kB/s | 26 kB 00:00 2026-02-22T21:27:56.830 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:27:56.830 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:27:56.849 INFO:teuthology.orchestra.run.trial140.stdout:(79/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.7 MB/s | 2.0 MB 00:00 2026-02-22T21:27:56.891 INFO:teuthology.orchestra.run.trial140.stdout:(80/99): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2026-02-22T21:27:56.904 INFO:teuthology.orchestra.run.trial051.stdout:(60/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.0 MB/s | 551 kB 00:00 2026-02-22T21:27:56.909 INFO:teuthology.orchestra.run.trial140.stdout:(81/99): python3-kubernetes-26.1.0-3.el9.noarch 7.9 MB/s | 1.0 MB 00:00 2026-02-22T21:27:56.912 INFO:teuthology.orchestra.run.trial140.stdout:(82/99): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2026-02-22T21:27:56.961 INFO:teuthology.orchestra.run.trial140.stdout:(83/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2026-02-22T21:27:56.968 INFO:teuthology.orchestra.run.trial051.stdout:(61/99): python3-autocommand-2.2.2-8.el9.noarch 139 kB/s | 29 kB 00:00 2026-02-22T21:27:56.972 INFO:teuthology.orchestra.run.trial140.stdout:(84/99): python3-natsort-7.1.1-5.el9.noarch.rpm 922 kB/s | 58 kB 00:00 2026-02-22T21:27:56.979 INFO:teuthology.orchestra.run.trial140.stdout:(85/99): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2026-02-22T21:27:57.023 INFO:teuthology.orchestra.run.trial140.stdout:(86/99): python3-portend-3.1.0-2.el9.noarch.rpm 265 kB/s | 16 kB 00:00 2026-02-22T21:27:57.035 INFO:teuthology.orchestra.run.trial068.stdout:(41/99): python3-libstoragemgmt-1.10.1-1.el9.x8 241 kB/s | 177 kB 00:00 2026-02-22T21:27:57.035 INFO:teuthology.orchestra.run.trial140.stdout:(87/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.4 MB/s | 90 kB 00:00 2026-02-22T21:27:57.041 INFO:teuthology.orchestra.run.trial140.stdout:(88/99): python3-repoze-lru-0.7-16.el9.noarch.r 496 kB/s | 31 kB 00:00 2026-02-22T21:27:57.097 INFO:teuthology.orchestra.run.trial140.stdout:(89/99): python3-rsa-4.9-2.el9.noarch.rpm 949 kB/s | 59 kB 00:00 2026-02-22T21:27:57.104 INFO:teuthology.orchestra.run.trial140.stdout:(90/99): python3-saml-1.16.0-1.el9.noarch.rpm 1.9 MB/s | 125 kB 00:00 2026-02-22T21:27:57.105 INFO:teuthology.orchestra.run.trial140.stdout:(91/99): python3-routes-2.5.1-5.el9.noarch.rpm 2.3 MB/s | 188 kB 00:00 2026-02-22T21:27:57.107 INFO:teuthology.orchestra.run.trial051.stdout:(62/99): python3-asyncssh-2.13.2-5.el9.noarch.r 1.1 MB/s | 548 kB 00:00 2026-02-22T21:27:57.159 INFO:teuthology.orchestra.run.trial140.stdout:(92/99): python3-tempora-5.0.0-2.el9.noarch.rpm 576 kB/s | 36 kB 00:00 2026-02-22T21:27:57.171 INFO:teuthology.orchestra.run.trial140.stdout:(93/99): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2026-02-22T21:27:57.174 INFO:teuthology.orchestra.run.trial140.stdout:(94/99): python3-typing-extensions-4.15.0-1.el9 1.2 MB/s | 86 kB 00:00 2026-02-22T21:27:57.180 INFO:teuthology.orchestra.run.trial051.stdout:(63/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 206 kB/s | 43 kB 00:00 2026-02-22T21:27:57.182 INFO:teuthology.orchestra.run.trial051.stdout:(64/99): python3-backports-tarfile-1.2.0-1.el9. 216 kB/s | 60 kB 00:00 2026-02-22T21:27:57.223 INFO:teuthology.orchestra.run.trial140.stdout:(95/99): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2026-02-22T21:27:57.239 INFO:teuthology.orchestra.run.trial140.stdout:(96/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 739 kB/s | 48 kB 00:00 2026-02-22T21:27:57.242 INFO:teuthology.orchestra.run.trial140.stdout:(97/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.9 MB/s | 427 kB 00:00 2026-02-22T21:27:57.248 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:27:57.248 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:27:57.286 INFO:teuthology.orchestra.run.trial140.stdout:(98/99): python3-xmltodict-0.12.0-15.el9.noarch 351 kB/s | 22 kB 00:00 2026-02-22T21:27:57.303 INFO:teuthology.orchestra.run.trial140.stdout:(99/99): python3-zc-lockfile-2.0-10.el9.noarch. 315 kB/s | 20 kB 00:00 2026-02-22T21:27:57.306 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:27:57.306 INFO:teuthology.orchestra.run.trial140.stdout:Total 14 MB/s | 95 MB 00:07 2026-02-22T21:27:57.316 INFO:teuthology.orchestra.run.trial051.stdout:(65/99): python3-cachetools-4.2.4-1.el9.noarch. 154 kB/s | 32 kB 00:00 2026-02-22T21:27:57.320 INFO:teuthology.orchestra.run.trial051.stdout:(66/99): python3-certifi-2023.05.07-4.el9.noarc 101 kB/s | 14 kB 00:00 2026-02-22T21:27:57.394 INFO:teuthology.orchestra.run.trial068.stdout:(42/99): python3-mako-1.1.4-6.el9.noarch.rpm 236 kB/s | 172 kB 00:00 2026-02-22T21:27:57.459 INFO:teuthology.orchestra.run.trial068.stdout:(43/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.3 MB/s | 1.2 MB 00:00 2026-02-22T21:27:57.541 INFO:teuthology.orchestra.run.trial051.stdout:(67/99): python3-cheroot-10.0.1-4.el9.noarch.rp 484 kB/s | 173 kB 00:00 2026-02-22T21:27:57.598 INFO:teuthology.orchestra.run.trial068.stdout:(44/99): python3-markupsafe-1.1.1-12.el9.x86_64 62 kB/s | 35 kB 00:00 2026-02-22T21:27:57.723 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:27:57.725 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-22T21:27:57.741 INFO:teuthology.orchestra.run.trial051.stdout:(68/99): python3-google-auth-2.45.0-1.el9.noarc 604 kB/s | 254 kB 00:00 2026-02-22T21:27:57.746 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:27:57.747 INFO:teuthology.orchestra.run.trial051.stdout:(69/99): python3-cherrypy-18.6.1-2.el9.noarch.r 831 kB/s | 358 kB 00:00 2026-02-22T21:27:57.756 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-22T21:27:57.761 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-22T21:27:57.782 INFO:teuthology.orchestra.run.trial125.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-22T21:27:57.784 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:27:57.784 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:27:57.789 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 5/99 2026-02-22T21:27:57.822 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-22T21:27:57.831 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-22T21:27:57.836 INFO:teuthology.orchestra.run.trial125.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-22T21:27:57.839 INFO:teuthology.orchestra.run.trial125.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-22T21:27:57.843 INFO:teuthology.orchestra.run.trial125.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-22T21:27:57.866 INFO:teuthology.orchestra.run.trial125.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-22T21:27:57.879 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-22T21:27:57.882 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-22T21:27:57.888 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-22T21:27:57.890 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-22T21:27:57.901 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-22T21:27:57.907 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-22T21:27:57.932 INFO:teuthology.orchestra.run.trial125.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-22T21:27:57.935 INFO:teuthology.orchestra.run.trial125.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-22T21:27:57.942 INFO:teuthology.orchestra.run.trial125.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-22T21:27:57.966 INFO:teuthology.orchestra.run.trial125.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-22T21:27:58.017 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-22T21:27:58.030 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 23/99 2026-02-22T21:27:58.032 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-22T21:27:58.037 INFO:teuthology.orchestra.run.trial125.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-22T21:27:58.042 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-22T21:27:58.047 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-22T21:27:58.050 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-22T21:27:58.064 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-22T21:27:58.085 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-22T21:27:58.090 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-22T21:27:58.095 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-22T21:27:58.099 INFO:teuthology.orchestra.run.trial051.stdout:(70/99): python3-grpcio-tools-1.46.7-10.el9.x86 403 kB/s | 144 kB 00:00 2026-02-22T21:27:58.107 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-22T21:27:58.109 INFO:teuthology.orchestra.run.trial051.stdout:(71/99): python3-influxdb-5.3.1-1.el9.noarch.rp 384 kB/s | 139 kB 00:00 2026-02-22T21:27:58.115 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-22T21:27:58.121 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-22T21:27:58.141 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-22T21:27:58.150 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-22T21:27:58.202 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:27:58.203 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:27:58.204 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-22T21:27:58.216 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-22T21:27:58.232 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-22T21:27:58.233 INFO:teuthology.orchestra.run.trial068.stdout:(45/99): python3-packaging-20.9-5.el9.noarch.rp 121 kB/s | 77 kB 00:00 2026-02-22T21:27:58.236 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-22T21:27:58.244 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-22T21:27:58.249 INFO:teuthology.orchestra.run.trial051.stdout:(72/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 76 kB/s | 11 kB 00:00 2026-02-22T21:27:58.264 INFO:teuthology.orchestra.run.trial051.stdout:(73/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2026-02-22T21:27:58.284 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-22T21:27:58.287 INFO:teuthology.orchestra.run.trial068.stdout:(46/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 534 kB/s | 442 kB 00:00 2026-02-22T21:27:58.389 INFO:teuthology.orchestra.run.trial051.stdout:(74/99): python3-isodate-0.6.1-3.el9.noarch.rpm 193 kB/s | 56 kB 00:00 2026-02-22T21:27:58.483 INFO:teuthology.orchestra.run.trial051.stdout:(75/99): python3-jaraco-classes-3.2.1-5.el9.noa 75 kB/s | 18 kB 00:00 2026-02-22T21:27:58.483 INFO:teuthology.orchestra.run.trial051.stdout:(76/99): python3-jaraco-collections-3.0.0-8.el9 105 kB/s | 23 kB 00:00 2026-02-22T21:27:58.522 INFO:teuthology.orchestra.run.trial068.stdout:(47/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.4 MB/s | 6.1 MB 00:01 2026-02-22T21:27:58.606 INFO:teuthology.orchestra.run.trial051.stdout:(77/99): python3-jaraco-context-6.0.1-3.el9.noa 90 kB/s | 20 kB 00:00 2026-02-22T21:27:58.608 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-22T21:27:58.619 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-22T21:27:58.622 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-22T21:27:58.628 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-22T21:27:58.630 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-22T21:27:58.658 INFO:teuthology.orchestra.run.trial125.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-22T21:27:58.668 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:27:58.670 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-22T21:27:58.698 INFO:teuthology.orchestra.run.trial051.stdout:(78/99): python3-jaraco-functools-3.5.0-2.el9.n 90 kB/s | 19 kB 00:00 2026-02-22T21:27:58.700 INFO:teuthology.orchestra.run.trial051.stdout:(79/99): python3-jaraco-text-4.0.0-2.el9.noarch 122 kB/s | 26 kB 00:00 2026-02-22T21:27:58.701 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-22T21:27:58.702 INFO:teuthology.orchestra.run.trial125.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-22T21:27:58.706 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-22T21:27:58.708 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-22T21:27:58.712 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-22T21:27:58.718 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-22T21:27:58.721 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-22T21:27:58.728 INFO:teuthology.orchestra.run.trial140.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-22T21:27:58.729 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-22T21:27:58.732 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-22T21:27:58.734 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 5/99 2026-02-22T21:27:58.759 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-22T21:27:58.767 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-22T21:27:58.769 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-22T21:27:58.776 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-22T21:27:58.781 INFO:teuthology.orchestra.run.trial140.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-22T21:27:58.784 INFO:teuthology.orchestra.run.trial140.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-22T21:27:58.788 INFO:teuthology.orchestra.run.trial140.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-22T21:27:58.804 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-22T21:27:58.811 INFO:teuthology.orchestra.run.trial140.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-22T21:27:58.824 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-22T21:27:58.827 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-22T21:27:58.833 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-22T21:27:58.835 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-22T21:27:58.846 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-22T21:27:58.852 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-22T21:27:58.877 INFO:teuthology.orchestra.run.trial140.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-22T21:27:58.880 INFO:teuthology.orchestra.run.trial140.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-22T21:27:58.887 INFO:teuthology.orchestra.run.trial140.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-22T21:27:58.911 INFO:teuthology.orchestra.run.trial140.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-22T21:27:58.962 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-22T21:27:58.975 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 23/99 2026-02-22T21:27:58.978 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-22T21:27:58.982 INFO:teuthology.orchestra.run.trial140.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-22T21:27:58.987 INFO:teuthology.orchestra.run.trial051.stdout:(80/99): python3-more-itertools-8.12.0-2.el9.no 275 kB/s | 79 kB 00:00 2026-02-22T21:27:58.987 INFO:teuthology.orchestra.run.trial051.stdout:(81/99): python3-logutils-0.3.5-21.el9.noarch.r 160 kB/s | 46 kB 00:00 2026-02-22T21:27:58.988 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-22T21:27:58.992 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-22T21:27:58.995 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-22T21:27:59.003 INFO:teuthology.orchestra.run.trial068.stdout:(48/99): python3-protobuf-3.14.0-17.el9.noarch. 347 kB/s | 267 kB 00:00 2026-02-22T21:27:59.010 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-22T21:27:59.020 INFO:teuthology.orchestra.run.trial068.stdout:(49/99): python3-pyasn1-0.4.8-6.el9.noarch.rpm 216 kB/s | 159 kB 00:00 2026-02-22T21:27:59.028 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-22T21:27:59.031 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-22T21:27:59.035 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-22T21:27:59.040 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-22T21:27:59.051 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-22T21:27:59.052 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-22T21:27:59.055 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-22T21:27:59.061 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-22T21:27:59.067 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-22T21:27:59.086 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-22T21:27:59.095 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-22T21:27:59.114 INFO:teuthology.orchestra.run.trial125.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-22T21:27:59.116 INFO:teuthology.orchestra.run.trial125.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-22T21:27:59.137 INFO:teuthology.orchestra.run.trial125.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-22T21:27:59.150 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-22T21:27:59.162 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-22T21:27:59.178 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-22T21:27:59.181 INFO:teuthology.orchestra.run.trial051.stdout:(82/99): python3-kubernetes-26.1.0-3.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2026-02-22T21:27:59.182 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-22T21:27:59.190 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-22T21:27:59.230 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-22T21:27:59.265 INFO:teuthology.orchestra.run.trial051.stdout:(83/99): python3-natsort-7.1.1-5.el9.noarch.rpm 208 kB/s | 58 kB 00:00 2026-02-22T21:27:59.265 INFO:teuthology.orchestra.run.trial051.stdout:(84/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 310 kB/s | 86 kB 00:00 2026-02-22T21:27:59.314 INFO:teuthology.orchestra.run.trial068.stdout:(50/99): python3-pyasn1-modules-0.4.8-6.el9.noa 353 kB/s | 279 kB 00:00 2026-02-22T21:27:59.462 INFO:teuthology.orchestra.run.trial125.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-22T21:27:59.482 INFO:teuthology.orchestra.run.trial051.stdout:(85/99): python3-portend-3.1.0-2.el9.noarch.rpm 76 kB/s | 16 kB 00:00 2026-02-22T21:27:59.535 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-22T21:27:59.545 INFO:teuthology.orchestra.run.trial051.stdout:(86/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 322 kB/s | 90 kB 00:00 2026-02-22T21:27:59.554 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-22T21:27:59.564 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-22T21:27:59.568 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-22T21:27:59.574 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-22T21:27:59.576 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-22T21:27:59.598 INFO:teuthology.orchestra.run.trial068.stdout:(51/99): python3-requests-oauthlib-1.3.0-12.el9 90 kB/s | 54 kB 00:00 2026-02-22T21:27:59.604 INFO:teuthology.orchestra.run.trial140.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-22T21:27:59.615 INFO:teuthology.orchestra.run.trial051.stdout:(87/99): python3-pecan-1.4.2-3.el9.noarch.rpm 627 kB/s | 272 kB 00:00 2026-02-22T21:27:59.649 INFO:teuthology.orchestra.run.trial140.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-22T21:27:59.656 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-22T21:27:59.660 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-22T21:27:59.665 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-22T21:27:59.669 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-22T21:27:59.676 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-22T21:27:59.679 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-22T21:27:59.693 INFO:teuthology.orchestra.run.trial051.stdout:(88/99): python3-repoze-lru-0.7-16.el9.noarch.r 146 kB/s | 31 kB 00:00 2026-02-22T21:27:59.707 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-22T21:27:59.717 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-22T21:27:59.752 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-22T21:27:59.891 INFO:teuthology.orchestra.run.trial068.stdout:(52/99): python3-toml-0.10.2-6.el9.noarch.rpm 72 kB/s | 42 kB 00:00 2026-02-22T21:27:59.901 INFO:teuthology.orchestra.run.trial051.stdout:(89/99): python3-rsa-4.9-2.el9.noarch.rpm 206 kB/s | 59 kB 00:00 2026-02-22T21:27:59.908 INFO:teuthology.orchestra.run.trial051.stdout:(90/99): python3-routes-2.5.1-5.el9.noarch.rpm 519 kB/s | 188 kB 00:00 2026-02-22T21:27:59.975 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-22T21:27:59.999 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-22T21:28:00.004 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-22T21:28:00.044 INFO:teuthology.orchestra.run.trial051.stdout:(91/99): python3-saml-1.16.0-1.el9.noarch.rpm 356 kB/s | 125 kB 00:00 2026-02-22T21:28:00.063 INFO:teuthology.orchestra.run.trial140.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-22T21:28:00.065 INFO:teuthology.orchestra.run.trial140.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-22T21:28:00.087 INFO:teuthology.orchestra.run.trial140.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-22T21:28:00.119 INFO:teuthology.orchestra.run.trial051.stdout:(92/99): python3-tempora-5.0.0-2.el9.noarch.rpm 164 kB/s | 36 kB 00:00 2026-02-22T21:28:00.186 INFO:teuthology.orchestra.run.trial051.stdout:(93/99): python3-typing-extensions-4.15.0-1.el9 311 kB/s | 86 kB 00:00 2026-02-22T21:28:00.210 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-22T21:28:00.227 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-22T21:28:00.237 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-22T21:28:00.243 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-22T21:28:00.257 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-22T21:28:00.272 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-22T21:28:00.331 INFO:teuthology.orchestra.run.trial068.stdout:(53/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 257 kB/s | 189 kB 00:00 2026-02-22T21:28:00.348 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-22T21:28:00.357 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-22T21:28:00.380 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-22T21:28:00.400 INFO:teuthology.orchestra.run.trial051.stdout:(94/99): python3-websocket-client-1.2.3-2.el9.n 319 kB/s | 90 kB 00:00 2026-02-22T21:28:00.407 INFO:teuthology.orchestra.run.trial068.stdout:(54/99): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2026-02-22T21:28:00.410 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-22T21:28:00.415 INFO:teuthology.orchestra.run.trial140.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-22T21:28:00.434 INFO:teuthology.orchestra.run.trial068.stdout:(55/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-22T21:28:00.460 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-22T21:28:00.466 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-22T21:28:00.472 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-22T21:28:00.473 INFO:teuthology.orchestra.run.trial051.stdout:(95/99): python3-webob-1.8.8-2.el9.noarch.rpm 538 kB/s | 230 kB 00:00 2026-02-22T21:28:00.473 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-22T21:28:00.488 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-22T21:28:00.491 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:00.491 INFO:teuthology.orchestra.run.trial125.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-22T21:28:00.491 INFO:teuthology.orchestra.run.trial125.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-22T21:28:00.491 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:00.500 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:00.527 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:00.527 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-22T21:28:00.527 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:00.550 INFO:teuthology.orchestra.run.trial068.stdout:(56/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 4.7 MB/s | 551 kB 00:00 2026-02-22T21:28:00.551 INFO:teuthology.orchestra.run.trial068.stdout:(57/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 137 kB/s | 90 kB 00:00 2026-02-22T21:28:00.558 INFO:teuthology.orchestra.run.trial068.stdout:(58/99): protobuf-compiler-3.14.0-17.el9.x86_64 5.6 MB/s | 862 kB 00:00 2026-02-22T21:28:00.563 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-22T21:28:00.566 INFO:teuthology.orchestra.run.trial068.stdout:(59/99): grpc-data-1.46.7-10.el9.noarch.rpm 1.3 MB/s | 19 kB 00:00 2026-02-22T21:28:00.580 INFO:teuthology.orchestra.run.trial068.stdout:(60/99): python3-autocommand-2.2.2-8.el9.noarch 1.9 MB/s | 29 kB 00:00 2026-02-22T21:28:00.596 INFO:teuthology.orchestra.run.trial068.stdout:(61/99): python3-backports-tarfile-1.2.0-1.el9. 3.8 MB/s | 60 kB 00:00 2026-02-22T21:28:00.604 INFO:teuthology.orchestra.run.trial051.stdout:(96/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.0 MB/s | 427 kB 00:00 2026-02-22T21:28:00.611 INFO:teuthology.orchestra.run.trial068.stdout:(62/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.8 MB/s | 43 kB 00:00 2026-02-22T21:28:00.626 INFO:teuthology.orchestra.run.trial068.stdout:(63/99): python3-cachetools-4.2.4-1.el9.noarch. 2.1 MB/s | 32 kB 00:00 2026-02-22T21:28:00.627 INFO:teuthology.orchestra.run.trial125.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-22T21:28:00.627 INFO:teuthology.orchestra.run.trial068.stdout:(64/99): luarocks-3.9.2-5.el9.noarch.rpm 1.9 MB/s | 151 kB 00:00 2026-02-22T21:28:00.629 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:00.641 INFO:teuthology.orchestra.run.trial068.stdout:(65/99): python3-certifi-2023.05.07-4.el9.noarc 964 kB/s | 14 kB 00:00 2026-02-22T21:28:00.643 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:00.643 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:00.643 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-22T21:28:00.643 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:00.656 INFO:teuthology.orchestra.run.trial068.stdout:(66/99): python3-cheroot-10.0.1-4.el9.noarch.rp 6.0 MB/s | 173 kB 00:00 2026-02-22T21:28:00.661 INFO:teuthology.orchestra.run.trial068.stdout:(67/99): python3-asyncssh-2.13.2-5.el9.noarch.r 5.2 MB/s | 548 kB 00:00 2026-02-22T21:28:00.662 INFO:teuthology.orchestra.run.trial068.stdout:(68/99): python3-cherrypy-18.6.1-2.el9.noarch.r 17 MB/s | 358 kB 00:00 2026-02-22T21:28:00.676 INFO:teuthology.orchestra.run.trial068.stdout:(69/99): python3-google-auth-2.45.0-1.el9.noarc 12 MB/s | 254 kB 00:00 2026-02-22T21:28:00.680 INFO:teuthology.orchestra.run.trial068.stdout:(70/99): python3-grpcio-tools-1.46.7-10.el9.x86 7.7 MB/s | 144 kB 00:00 2026-02-22T21:28:00.683 INFO:teuthology.orchestra.run.trial051.stdout:(97/99): python3-xmltodict-0.12.0-15.el9.noarch 105 kB/s | 22 kB 00:00 2026-02-22T21:28:00.694 INFO:teuthology.orchestra.run.trial068.stdout:(71/99): python3-influxdb-5.3.1-1.el9.noarch.rp 7.9 MB/s | 139 kB 00:00 2026-02-22T21:28:00.696 INFO:teuthology.orchestra.run.trial068.stdout:(72/99): python3-isodate-0.6.1-3.el9.noarch.rpm 3.4 MB/s | 56 kB 00:00 2026-02-22T21:28:00.703 INFO:teuthology.orchestra.run.trial051.stdout:(98/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 160 kB/s | 48 kB 00:00 2026-02-22T21:28:00.708 INFO:teuthology.orchestra.run.trial068.stdout:(73/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 729 kB/s | 11 kB 00:00 2026-02-22T21:28:00.712 INFO:teuthology.orchestra.run.trial068.stdout:(74/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 40 MB/s | 2.0 MB 00:00 2026-02-22T21:28:00.712 INFO:teuthology.orchestra.run.trial068.stdout:(75/99): python3-jaraco-classes-3.2.1-5.el9.noa 1.1 MB/s | 18 kB 00:00 2026-02-22T21:28:00.723 INFO:teuthology.orchestra.run.trial068.stdout:(76/99): python3-jaraco-collections-3.0.0-8.el9 1.5 MB/s | 23 kB 00:00 2026-02-22T21:28:00.726 INFO:teuthology.orchestra.run.trial068.stdout:(77/99): python3-jaraco-context-6.0.1-3.el9.noa 1.3 MB/s | 20 kB 00:00 2026-02-22T21:28:00.727 INFO:teuthology.orchestra.run.trial068.stdout:(78/99): python3-jaraco-functools-3.5.0-2.el9.n 1.2 MB/s | 19 kB 00:00 2026-02-22T21:28:00.738 INFO:teuthology.orchestra.run.trial068.stdout:(79/99): python3-jaraco-text-4.0.0-2.el9.noarch 1.7 MB/s | 26 kB 00:00 2026-02-22T21:28:00.744 INFO:teuthology.orchestra.run.trial068.stdout:(80/99): python3-logutils-0.3.5-21.el9.noarch.r 2.8 MB/s | 46 kB 00:00 2026-02-22T21:28:00.746 INFO:teuthology.orchestra.run.trial068.stdout:(81/99): python3-kubernetes-26.1.0-3.el9.noarch 53 MB/s | 1.0 MB 00:00 2026-02-22T21:28:00.754 INFO:teuthology.orchestra.run.trial068.stdout:(82/99): python3-more-itertools-8.12.0-2.el9.no 4.9 MB/s | 79 kB 00:00 2026-02-22T21:28:00.759 INFO:teuthology.orchestra.run.trial068.stdout:(83/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 5.3 MB/s | 86 kB 00:00 2026-02-22T21:28:00.761 INFO:teuthology.orchestra.run.trial068.stdout:(84/99): python3-natsort-7.1.1-5.el9.noarch.rpm 3.8 MB/s | 58 kB 00:00 2026-02-22T21:28:00.774 INFO:teuthology.orchestra.run.trial068.stdout:(85/99): python3-portend-3.1.0-2.el9.noarch.rpm 1.1 MB/s | 16 kB 00:00 2026-02-22T21:28:00.775 INFO:teuthology.orchestra.run.trial068.stdout:(86/99): python3-pecan-1.4.2-3.el9.noarch.rpm 13 MB/s | 272 kB 00:00 2026-02-22T21:28:00.776 INFO:teuthology.orchestra.run.trial068.stdout:(87/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 5.7 MB/s | 90 kB 00:00 2026-02-22T21:28:00.790 INFO:teuthology.orchestra.run.trial068.stdout:(88/99): python3-repoze-lru-0.7-16.el9.noarch.r 2.0 MB/s | 31 kB 00:00 2026-02-22T21:28:00.792 INFO:teuthology.orchestra.run.trial068.stdout:(89/99): python3-rsa-4.9-2.el9.noarch.rpm 3.8 MB/s | 59 kB 00:00 2026-02-22T21:28:00.793 INFO:teuthology.orchestra.run.trial068.stdout:(90/99): python3-routes-2.5.1-5.el9.noarch.rpm 10 MB/s | 188 kB 00:00 2026-02-22T21:28:00.807 INFO:teuthology.orchestra.run.trial068.stdout:(91/99): python3-saml-1.16.0-1.el9.noarch.rpm 7.3 MB/s | 125 kB 00:00 2026-02-22T21:28:00.807 INFO:teuthology.orchestra.run.trial068.stdout:(92/99): python3-tempora-5.0.0-2.el9.noarch.rpm 2.3 MB/s | 36 kB 00:00 2026-02-22T21:28:00.809 INFO:teuthology.orchestra.run.trial068.stdout:(93/99): python3-typing-extensions-4.15.0-1.el9 5.4 MB/s | 86 kB 00:00 2026-02-22T21:28:00.819 INFO:teuthology.orchestra.run.trial051.stdout:(99/99): python3-zc-lockfile-2.0-10.el9.noarch. 93 kB/s | 20 kB 00:00 2026-02-22T21:28:00.823 INFO:teuthology.orchestra.run.trial068.stdout:(94/99): python3-websocket-client-1.2.3-2.el9.n 5.6 MB/s | 90 kB 00:00 2026-02-22T21:28:00.825 INFO:teuthology.orchestra.run.trial068.stdout:(95/99): python3-webob-1.8.8-2.el9.noarch.rpm 13 MB/s | 230 kB 00:00 2026-02-22T21:28:00.827 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:00.827 INFO:teuthology.orchestra.run.trial051.stdout:Total 11 MB/s | 95 MB 00:08 2026-02-22T21:28:00.834 INFO:teuthology.orchestra.run.trial068.stdout:(96/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 17 MB/s | 427 kB 00:00 2026-02-22T21:28:00.838 INFO:teuthology.orchestra.run.trial068.stdout:(97/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 3.2 MB/s | 48 kB 00:00 2026-02-22T21:28:00.840 INFO:teuthology.orchestra.run.trial068.stdout:(98/99): python3-xmltodict-0.12.0-15.el9.noarch 1.4 MB/s | 22 kB 00:00 2026-02-22T21:28:00.849 INFO:teuthology.orchestra.run.trial068.stdout:(99/99): python3-zc-lockfile-2.0-10.el9.noarch. 1.3 MB/s | 20 kB 00:00 2026-02-22T21:28:00.850 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:00.851 INFO:teuthology.orchestra.run.trial068.stdout:Total 11 MB/s | 95 MB 00:08 2026-02-22T21:28:01.160 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-22T21:28:01.177 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-22T21:28:01.187 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-22T21:28:01.193 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-22T21:28:01.207 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-22T21:28:01.223 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-22T21:28:01.252 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:01.279 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:01.290 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:01.290 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:01.298 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-22T21:28:01.307 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-22T21:28:01.316 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:01.316 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:01.330 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-22T21:28:01.360 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-22T21:28:01.367 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:01.392 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:01.410 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-22T21:28:01.417 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-22T21:28:01.421 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-22T21:28:01.422 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-22T21:28:01.441 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:01.441 INFO:teuthology.orchestra.run.trial140.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-22T21:28:01.441 INFO:teuthology.orchestra.run.trial140.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-22T21:28:01.441 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:01.450 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:01.452 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:01.454 INFO:teuthology.orchestra.run.trial125.stdout: Installing : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:01.459 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 88/99 2026-02-22T21:28:01.476 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:01.476 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-22T21:28:01.476 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:01.479 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 89/99 2026-02-22T21:28:01.481 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:01.511 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-22T21:28:01.574 INFO:teuthology.orchestra.run.trial140.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-22T21:28:01.576 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:01.590 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:01.590 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:01.590 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-22T21:28:01.591 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:01.708 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:01.708 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:01.737 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:01.737 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:01.981 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:01.984 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:02.171 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:02.173 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-22T21:28:02.201 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:02.203 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-22T21:28:02.203 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-22T21:28:02.209 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-22T21:28:02.230 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-22T21:28:02.234 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-22T21:28:02.236 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 5/99 2026-02-22T21:28:02.239 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-22T21:28:02.261 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-22T21:28:02.267 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 5/99 2026-02-22T21:28:02.270 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-22T21:28:02.280 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-22T21:28:02.285 INFO:teuthology.orchestra.run.trial051.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-22T21:28:02.287 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-22T21:28:02.292 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-22T21:28:02.300 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-22T21:28:02.310 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-22T21:28:02.312 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:02.315 INFO:teuthology.orchestra.run.trial051.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-22T21:28:02.315 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-22T21:28:02.318 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-22T21:28:02.322 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-22T21:28:02.328 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-22T21:28:02.331 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-22T21:28:02.337 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:02.337 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:02.337 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-22T21:28:02.338 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:02.338 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:02.338 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:02.339 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-22T21:28:02.339 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-22T21:28:02.345 INFO:teuthology.orchestra.run.trial068.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-22T21:28:02.350 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-22T21:28:02.356 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-22T21:28:02.358 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-22T21:28:02.362 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-22T21:28:02.368 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-22T21:28:02.369 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-22T21:28:02.380 INFO:teuthology.orchestra.run.trial051.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-22T21:28:02.381 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-22T21:28:02.383 INFO:teuthology.orchestra.run.trial051.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-22T21:28:02.387 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-22T21:28:02.390 INFO:teuthology.orchestra.run.trial051.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-22T21:28:02.397 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:02.399 INFO:teuthology.orchestra.run.trial140.stdout: Installing : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:02.403 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 88/99 2026-02-22T21:28:02.411 INFO:teuthology.orchestra.run.trial068.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-22T21:28:02.414 INFO:teuthology.orchestra.run.trial051.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-22T21:28:02.414 INFO:teuthology.orchestra.run.trial068.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-22T21:28:02.421 INFO:teuthology.orchestra.run.trial068.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-22T21:28:02.423 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 89/99 2026-02-22T21:28:02.425 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:02.446 INFO:teuthology.orchestra.run.trial068.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-22T21:28:02.447 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:02.448 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:02.461 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:02.461 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:02.465 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-22T21:28:02.477 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 23/99 2026-02-22T21:28:02.482 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-22T21:28:02.489 INFO:teuthology.orchestra.run.trial051.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-22T21:28:02.494 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-22T21:28:02.497 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-22T21:28:02.498 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-22T21:28:02.502 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-22T21:28:02.510 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 23/99 2026-02-22T21:28:02.512 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-22T21:28:02.516 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-22T21:28:02.517 INFO:teuthology.orchestra.run.trial068.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-22T21:28:02.517 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:02.522 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-22T21:28:02.527 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-22T21:28:02.530 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-22T21:28:02.536 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-22T21:28:02.541 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-22T21:28:02.544 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-22T21:28:02.546 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-22T21:28:02.557 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-22T21:28:02.565 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-22T21:28:02.566 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-22T21:28:02.567 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 94/99 2026-02-22T21:28:02.568 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:02.570 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-22T21:28:02.572 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-22T21:28:02.575 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-22T21:28:02.586 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:02.591 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-22T21:28:02.595 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-22T21:28:02.600 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-22T21:28:02.601 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-22T21:28:02.602 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:02.621 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-22T21:28:02.630 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-22T21:28:02.654 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-22T21:28:02.667 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-22T21:28:02.682 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-22T21:28:02.724 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-22T21:28:02.724 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-22T21:28:02.724 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-22T21:28:02.724 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-22T21:28:02.725 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-22T21:28:02.725 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-22T21:28:02.725 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-22T21:28:02.734 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-22T21:28:02.765 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-22T21:28:02.832 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:02.833 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:02.854 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:02.922 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:02.926 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:02.961 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:02.981 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:02.981 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:02.982 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-22T21:28:02.982 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:02.982 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:02.982 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:03.000 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:03.055 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-22T21:28:03.065 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-22T21:28:03.069 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-22T21:28:03.075 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-22T21:28:03.077 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-22T21:28:03.089 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-22T21:28:03.099 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-22T21:28:03.103 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-22T21:28:03.104 INFO:teuthology.orchestra.run.trial051.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-22T21:28:03.109 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-22T21:28:03.111 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-22T21:28:03.139 INFO:teuthology.orchestra.run.trial068.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-22T21:28:03.149 INFO:teuthology.orchestra.run.trial051.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-22T21:28:03.155 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-22T21:28:03.159 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-22T21:28:03.164 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-22T21:28:03.168 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-22T21:28:03.175 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-22T21:28:03.178 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-22T21:28:03.184 INFO:teuthology.orchestra.run.trial068.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-22T21:28:03.190 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-22T21:28:03.194 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-22T21:28:03.200 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-22T21:28:03.203 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-22T21:28:03.206 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-22T21:28:03.210 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-22T21:28:03.214 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-22T21:28:03.216 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-22T21:28:03.241 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-22T21:28:03.250 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-22T21:28:03.251 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-22T21:28:03.286 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-22T21:28:03.401 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:03.402 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:03.415 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:03.415 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:03.471 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:03.474 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-22T21:28:03.497 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-22T21:28:03.501 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-22T21:28:03.511 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-22T21:28:03.520 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 94/99 2026-02-22T21:28:03.521 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:03.535 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-22T21:28:03.539 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:03.544 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:03.556 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:03.560 INFO:teuthology.orchestra.run.trial051.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-22T21:28:03.562 INFO:teuthology.orchestra.run.trial051.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-22T21:28:03.583 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-22T21:28:03.598 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-22T21:28:03.599 INFO:teuthology.orchestra.run.trial068.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-22T21:28:03.622 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-22T21:28:03.651 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 1/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 3/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 4/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 5/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 6/99 2026-02-22T21:28:03.652 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 7/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 8/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 9/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 10/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 11/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 12/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 13/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 14/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 15/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-22T21:28:03.653 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-22T21:28:03.654 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-22T21:28:03.655 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-22T21:28:03.656 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-22T21:28:03.657 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-22T21:28:03.658 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-22T21:28:03.659 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-22T21:28:03.660 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-22T21:28:03.714 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-22T21:28:03.714 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:03.714 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.715 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-22T21:28:03.716 INFO:teuthology.orchestra.run.trial125.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: lua-5.4.4-4.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-22T21:28:03.717 INFO:teuthology.orchestra.run.trial125.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-22T21:28:03.718 INFO:teuthology.orchestra.run.trial125.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-22T21:28:03.719 INFO:teuthology.orchestra.run.trial125.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-22T21:28:03.720 INFO:teuthology.orchestra.run.trial125.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-ply-3.11-14.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-22T21:28:03.721 INFO:teuthology.orchestra.run.trial125.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-22T21:28:03.722 INFO:teuthology.orchestra.run.trial125.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: unzip-6.0-59.el9.x86_64 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: zip-3.0-35.el9.x86_64 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:03.723 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:03.768 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-base 2026-02-22T21:28:03.778 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:03.779 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:03.802 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:03.802 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:03.802 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-22T21:28:03.803 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:03.803 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:03.803 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:03.907 INFO:teuthology.orchestra.run.trial051.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-22T21:28:03.908 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:03.930 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:03.948 INFO:teuthology.orchestra.run.trial068.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-22T21:28:03.950 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:03.979 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-22T21:28:04.021 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-22T21:28:04.080 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:35 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:04.146 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:04.164 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:04.164 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:04.164 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:04.185 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install cephadm 2026-02-22T21:28:04.496 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:35 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:04.561 INFO:teuthology.orchestra.run.trial125.stdout:Package cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:04.579 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:04.579 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:04.579 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 1/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 3/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 4/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 5/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 6/99 2026-02-22T21:28:04.581 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 7/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 8/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 9/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 10/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 11/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 12/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 13/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 14/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 15/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-22T21:28:04.582 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-22T21:28:04.583 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-22T21:28:04.584 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-22T21:28:04.587 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-22T21:28:04.588 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-22T21:28:04.601 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-immutable-object-cache 2026-02-22T21:28:04.652 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-22T21:28:04.652 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:04.652 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:04.652 INFO:teuthology.orchestra.run.trial140.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: lua-5.4.4-4.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-22T21:28:04.653 INFO:teuthology.orchestra.run.trial140.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-22T21:28:04.654 INFO:teuthology.orchestra.run.trial140.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-ply-3.11-14.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-22T21:28:04.655 INFO:teuthology.orchestra.run.trial140.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: unzip-6.0-59.el9.x86_64 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: zip-3.0-35.el9.x86_64 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:04.656 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-22T21:28:04.670 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-22T21:28:04.680 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-22T21:28:04.686 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-22T21:28:04.694 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-22T21:28:04.700 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-22T21:28:04.708 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-base 2026-02-22T21:28:04.711 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-22T21:28:04.715 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-22T21:28:04.721 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-22T21:28:04.728 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-22T21:28:04.741 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-22T21:28:04.757 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-22T21:28:04.791 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-22T21:28:04.799 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-22T21:28:04.823 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-22T21:28:04.833 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-22T21:28:04.841 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-22T21:28:04.853 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-22T21:28:04.865 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-22T21:28:04.895 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-22T21:28:04.903 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-22T21:28:04.907 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:35 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:04.909 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-22T21:28:04.913 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-22T21:28:04.914 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-22T21:28:04.931 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:04.931 INFO:teuthology.orchestra.run.trial051.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-22T21:28:04.931 INFO:teuthology.orchestra.run.trial051.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-22T21:28:04.932 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:04.939 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:04.945 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-22T21:28:04.951 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-22T21:28:04.955 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-22T21:28:04.956 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-22T21:28:04.965 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:04.965 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-22T21:28:04.965 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:04.975 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:04.975 INFO:teuthology.orchestra.run.trial068.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-22T21:28:04.975 INFO:teuthology.orchestra.run.trial068.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-22T21:28:04.975 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:04.982 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:04.991 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 145 k 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 145 k 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 431 k 2026-02-22T21:28:04.992 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:05.002 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-22T21:28:05.009 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T21:28:05.009 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-22T21:28:05.009 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:05.021 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:05.046 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-22T21:28:05.066 INFO:teuthology.orchestra.run.trial051.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-22T21:28:05.068 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:05.083 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:05.083 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:05.083 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-22T21:28:05.083 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:05.086 INFO:teuthology.orchestra.run.trial125.stdout:ceph-immutable-object-cache-19.2.3-689.gf90f4ea 1.5 MB/s | 145 kB 00:00 2026-02-22T21:28:05.086 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:05.086 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:05.086 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.5 MB/s | 145 kB 00:00 2026-02-22T21:28:05.086 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:05.090 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:05.090 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:05.104 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:05.105 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:05.105 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:05.110 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-22T21:28:05.112 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:05.121 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:05.122 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:05.126 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install cephadm 2026-02-22T21:28:05.126 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T21:28:05.127 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:05.127 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-22T21:28:05.127 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:05.176 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:05.177 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:05.201 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:05.201 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:05.201 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-22T21:28:05.201 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:05.434 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:05.498 INFO:teuthology.orchestra.run.trial140.stdout:Package cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:05.516 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:05.517 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:05.517 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:05.518 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:05.537 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-immutable-object-cache 2026-02-22T21:28:05.545 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr 2026-02-22T21:28:05.782 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:05.807 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:05.808 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:05.808 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-22T21:28:05.808 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:05.808 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:05.808 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:05.826 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:05.842 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T21:28:05.852 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:05.856 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:36 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:05.867 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:05.869 INFO:teuthology.orchestra.run.trial051.stdout: Installing : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:05.874 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 88/99 2026-02-22T21:28:05.894 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 89/99 2026-02-22T21:28:05.895 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:05.911 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:05.913 INFO:teuthology.orchestra.run.trial068.stdout: Installing : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T21:28:05.918 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 88/99 2026-02-22T21:28:05.921 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 145 k 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:05.926 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:05.927 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 145 k 2026-02-22T21:28:05.927 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 431 k 2026-02-22T21:28:05.927 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:05.938 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 89/99 2026-02-22T21:28:05.939 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:05.940 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:05.940 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:05.940 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:05.961 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr-dashboard 2026-02-22T21:28:06.014 INFO:teuthology.orchestra.run.trial140.stdout:ceph-immutable-object-cache-19.2.3-689.gf90f4ea 1.6 MB/s | 145 kB 00:00 2026-02-22T21:28:06.014 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:06.014 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.6 MB/s | 145 kB 00:00 2026-02-22T21:28:06.014 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:06.018 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:06.018 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:06.050 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:06.050 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:06.105 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:06.106 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:06.130 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:06.130 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:06.130 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-22T21:28:06.130 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:06.268 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:06.332 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:06.351 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:06.351 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:06.351 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:06.372 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-22T21:28:06.384 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:06.387 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:06.428 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T21:28:06.431 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:06.443 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:06.471 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr 2026-02-22T21:28:06.681 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:06.746 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:06.765 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:06.765 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:06.765 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:06.784 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:38 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:06.785 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr-rook 2026-02-22T21:28:06.839 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:06.840 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:06.849 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:06.853 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:06.854 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:06.867 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:06.868 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:06.868 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:06.886 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T21:28:06.887 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:06.889 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr-dashboard 2026-02-22T21:28:06.899 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T21:28:06.900 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:06.909 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:06.955 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T21:28:06.958 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 94/99 2026-02-22T21:28:06.959 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:06.981 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:06.982 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:06.982 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-22T21:28:06.982 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:06.982 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:06.982 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:06.994 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:07.004 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 94/99 2026-02-22T21:28:07.005 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T21:28:07.026 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:07.037 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:07.091 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:38 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:07.155 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:07.174 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:07.174 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:07.174 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:07.195 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr-cephadm 2026-02-22T21:28:07.196 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:07.209 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:07.211 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:07.231 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:07.232 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:07.232 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-22T21:28:07.232 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:07.232 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:07.232 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:07.257 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T21:28:07.258 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:07.261 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:07.280 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:07.301 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-22T21:28:07.333 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:07.354 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:07.373 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:07.382 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:07.403 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T21:28:07.403 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:07.404 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-22T21:28:07.404 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:07.404 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T21:28:07.404 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:07.422 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:07.502 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:38 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:07.567 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:07.585 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:07.585 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:07.585 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:07.607 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-fuse 2026-02-22T21:28:07.609 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:07.674 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:07.692 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:07.692 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:07.692 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:07.713 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr-rook 2026-02-22T21:28:07.916 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:38 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:07.999 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout: ceph-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.2 M 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 1.2 M 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 3.6 M 2026-02-22T21:28:08.000 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:08.022 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 1/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 3/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 4/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 5/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 6/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 7/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 8/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 9/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 10/99 2026-02-22T21:28:08.035 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 11/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 12/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 13/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 14/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 15/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-22T21:28:08.036 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-22T21:28:08.037 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-22T21:28:08.037 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-22T21:28:08.038 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-22T21:28:08.039 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-22T21:28:08.040 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-22T21:28:08.040 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-22T21:28:08.040 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-22T21:28:08.040 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-22T21:28:08.040 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-22T21:28:08.041 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-22T21:28:08.041 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-22T21:28:08.041 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-22T21:28:08.041 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-22T21:28:08.041 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-22T21:28:08.046 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-22T21:28:08.087 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 1/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 3/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 4/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 5/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 6/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 7/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 8/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 9/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 10/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 11/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 12/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 13/99 2026-02-22T21:28:08.093 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 14/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 15/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-22T21:28:08.094 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-22T21:28:08.095 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-22T21:28:08.095 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-22T21:28:08.098 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-22T21:28:08.099 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-22T21:28:08.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-22T21:28:08.101 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.105 INFO:teuthology.orchestra.run.trial051.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: lua-5.4.4-4.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-22T21:28:08.106 INFO:teuthology.orchestra.run.trial051.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-22T21:28:08.107 INFO:teuthology.orchestra.run.trial051.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-ply-3.11-14.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-22T21:28:08.108 INFO:teuthology.orchestra.run.trial051.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: unzip-6.0-59.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: zip-3.0-35.el9.x86_64 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:08.109 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:08.110 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:08.110 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:08.110 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:08.124 INFO:teuthology.orchestra.run.trial125.stdout:ceph-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 9.3 MB/s | 1.2 MB 00:00 2026-02-22T21:28:08.124 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:08.125 INFO:teuthology.orchestra.run.trial125.stdout:Total 9.2 MB/s | 1.2 MB 00:00 2026-02-22T21:28:08.125 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:08.126 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr-cephadm 2026-02-22T21:28:08.129 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:08.129 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:08.159 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-base 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-22T21:28:08.162 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: lua-5.4.4-4.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-22T21:28:08.163 INFO:teuthology.orchestra.run.trial068.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-ply-3.11-14.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: unzip-6.0-59.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: zip-3.0-35.el9.x86_64 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:08.164 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:08.165 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:08.165 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:08.221 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base 2026-02-22T21:28:08.271 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:08.288 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:08.438 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:08.473 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:08.503 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:08.503 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:08.522 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:08.523 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:08.523 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:08.530 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:08.538 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:08.545 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-fuse 2026-02-22T21:28:08.556 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:08.557 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:08.557 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:08.561 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:08.578 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install cephadm 2026-02-22T21:28:08.589 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-volume 2026-02-22T21:28:08.595 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:08.613 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:08.614 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:08.614 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:08.636 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install cephadm 2026-02-22T21:28:08.853 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:08.890 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:08.906 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:08.935 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout: ceph-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.2 M 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:08.936 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:08.937 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:08.937 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 1.2 M 2026-02-22T21:28:08.937 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 3.6 M 2026-02-22T21:28:08.937 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:08.942 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:37 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:08.954 INFO:teuthology.orchestra.run.trial051.stdout:Package cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:08.971 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:08.972 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:08.973 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:08.973 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:08.989 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:08.990 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:08.990 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:08.993 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-immutable-object-cache 2026-02-22T21:28:09.007 INFO:teuthology.orchestra.run.trial068.stdout:Package cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:09.011 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librados-devel 2026-02-22T21:28:09.025 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:09.026 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:09.026 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:09.047 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-immutable-object-cache 2026-02-22T21:28:09.057 INFO:teuthology.orchestra.run.trial140.stdout:ceph-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 9.6 MB/s | 1.2 MB 00:00 2026-02-22T21:28:09.057 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:09.057 INFO:teuthology.orchestra.run.trial140.stdout:Total 9.5 MB/s | 1.2 MB 00:00 2026-02-22T21:28:09.057 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:09.061 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:09.061 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:09.098 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:09.098 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:09.201 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:09.217 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.297 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:09.318 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:09.354 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:38 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:09.380 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:09.380 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 145 k 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 145 k 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 431 k 2026-02-22T21:28:09.381 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:09.400 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout: librados-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 126 k 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 126 k 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 456 k 2026-02-22T21:28:09.401 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:09.437 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 145 k 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:09.438 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:09.439 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:09.439 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 145 k 2026-02-22T21:28:09.439 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 431 k 2026-02-22T21:28:09.439 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:09.460 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.467 INFO:teuthology.orchestra.run.trial051.stdout:ceph-immutable-object-cache-19.2.3-689.gf90f4ea 1.6 MB/s | 145 kB 00:00 2026-02-22T21:28:09.467 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:09.467 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.6 MB/s | 145 kB 00:00 2026-02-22T21:28:09.467 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:09.472 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:09.472 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:09.487 INFO:teuthology.orchestra.run.trial125.stdout:librados-devel-19.2.3-689.gf90f4eae.el9.x86_64. 1.4 MB/s | 126 kB 00:00 2026-02-22T21:28:09.488 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:09.488 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-22T21:28:09.488 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:09.490 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:09.491 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:09.503 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:09.503 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:09.516 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.516 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:09.516 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:09.517 INFO:teuthology.orchestra.run.trial140.stdout: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:09.517 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:09.517 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:09.522 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:09.522 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:09.524 INFO:teuthology.orchestra.run.trial068.stdout:ceph-immutable-object-cache-19.2.3-689.gf90f4ea 1.7 MB/s | 145 kB 00:00 2026-02-22T21:28:09.525 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:09.525 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.6 MB/s | 145 kB 00:00 2026-02-22T21:28:09.525 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:09.528 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:09.529 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:09.545 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-volume 2026-02-22T21:28:09.557 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:09.559 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.560 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:09.560 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:09.578 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:09.580 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.580 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:09.580 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-22T21:28:09.580 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:09.594 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.615 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:09.616 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.638 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.638 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:09.638 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-22T21:28:09.639 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:09.674 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.731 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:09.731 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:09.732 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:09.732 INFO:teuthology.orchestra.run.trial125.stdout: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:09.732 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:09.732 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:09.760 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install libcephfs2 2026-02-22T21:28:09.854 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:09.868 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:09.896 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr 2026-02-22T21:28:09.918 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:09.937 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:09.938 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:09.938 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:09.942 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T21:28:09.942 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:09.943 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:09.943 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:09.943 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:09.943 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:09.959 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librados-devel 2026-02-22T21:28:09.970 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr 2026-02-22T21:28:10.075 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:10.139 INFO:teuthology.orchestra.run.trial125.stdout:Package libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:10.157 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:10.158 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:10.158 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:10.179 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install libcephfs-devel 2026-02-22T21:28:10.207 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:10.272 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:10.273 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:10.283 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:10.292 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:10.292 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:10.292 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:10.314 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-dashboard 2026-02-22T21:28:10.347 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:10.355 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout: librados-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 126 k 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 126 k 2026-02-22T21:28:10.356 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 456 k 2026-02-22T21:28:10.357 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:10.366 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:10.366 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:10.366 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:10.387 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-dashboard 2026-02-22T21:28:10.483 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:10.566 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 34 k 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 34 k 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 153 k 2026-02-22T21:28:10.567 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:10.624 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:10.690 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:10.696 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:39 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:10.708 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:10.709 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:10.709 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:10.730 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-22T21:28:10.760 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:10.773 INFO:teuthology.orchestra.run.trial140.stdout:librados-devel-19.2.3-689.gf90f4eae.el9.x86_64. 303 kB/s | 126 kB 00:00 2026-02-22T21:28:10.774 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:10.774 INFO:teuthology.orchestra.run.trial140.stdout:Total 302 kB/s | 126 kB 00:00 2026-02-22T21:28:10.774 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:10.777 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:10.777 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:10.778 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:10.779 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:10.779 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:10.800 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-22T21:28:10.810 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:10.810 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:10.843 INFO:teuthology.orchestra.run.trial125.stdout:libcephfs-devel-19.2.3-689.gf90f4eae.el9.x86_64 122 kB/s | 34 kB 00:00 2026-02-22T21:28:10.844 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:10.844 INFO:teuthology.orchestra.run.trial125.stdout:Total 121 kB/s | 34 kB 00:00 2026-02-22T21:28:10.844 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:10.846 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:10.846 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:10.850 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:10.850 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:10.865 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:10.871 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:10.881 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:10.887 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:10.946 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:10.962 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:11.005 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:11.005 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:11.005 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:11.005 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:11.006 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:11.006 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:11.019 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:11.033 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librados2 2026-02-22T21:28:11.037 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:11.045 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install libcephfs2 2026-02-22T21:28:11.102 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:11.106 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:11.121 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:11.121 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:11.122 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:11.142 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-rook 2026-02-22T21:28:11.171 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:11.189 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:11.190 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:11.190 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:11.210 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-rook 2026-02-22T21:28:11.343 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:11.354 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:11.408 INFO:teuthology.orchestra.run.trial125.stdout:Package librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:11.418 INFO:teuthology.orchestra.run.trial140.stdout:Package libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:11.426 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:11.426 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:11.426 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:11.437 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:11.437 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:11.437 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:11.447 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librbd1 2026-02-22T21:28:11.449 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:11.458 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install libcephfs-devel 2026-02-22T21:28:11.512 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:11.513 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:11.532 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:11.532 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:11.532 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:11.553 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-cephadm 2026-02-22T21:28:11.577 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:11.595 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:11.595 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:11.595 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:11.617 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-cephadm 2026-02-22T21:28:11.757 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:11.768 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:11.822 INFO:teuthology.orchestra.run.trial125.stdout:Package librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:11.840 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:11.840 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:11.840 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 34 k 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:11.852 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:11.853 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:11.853 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 34 k 2026-02-22T21:28:11.853 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 153 k 2026-02-22T21:28:11.853 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:11.860 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:11.861 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rados 2026-02-22T21:28:11.918 INFO:teuthology.orchestra.run.trial140.stdout:libcephfs-devel-19.2.3-689.gf90f4eae.el9.x86_64 520 kB/s | 34 kB 00:00 2026-02-22T21:28:11.918 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:11.918 INFO:teuthology.orchestra.run.trial140.stdout:Total 514 kB/s | 34 kB 00:00 2026-02-22T21:28:11.919 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:11.920 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:11.920 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:11.923 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:11.923 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:11.923 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:40 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:11.925 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:11.943 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:11.944 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:11.944 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:11.945 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:11.960 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:11.966 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-fuse 2026-02-22T21:28:11.988 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:12.006 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:12.006 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:12.006 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:12.016 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.027 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-fuse 2026-02-22T21:28:12.073 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.074 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:12.074 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:12.074 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:12.074 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:12.074 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:12.101 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librados2 2026-02-22T21:28:12.172 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:12.237 INFO:teuthology.orchestra.run.trial125.stdout:Package python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:12.256 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:12.256 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:12.256 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:12.276 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:12.278 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rgw 2026-02-22T21:28:12.332 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:41 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:12.360 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.2 M 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:12.361 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:12.362 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 1.2 M 2026-02-22T21:28:12.362 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 3.6 M 2026-02-22T21:28:12.362 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:12.414 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:12.414 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.2 M 2026-02-22T21:28:12.415 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 1.2 M 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 3.6 M 2026-02-22T21:28:12.416 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:12.479 INFO:teuthology.orchestra.run.trial140.stdout:Package librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:12.481 INFO:teuthology.orchestra.run.trial051.stdout:ceph-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 9.6 MB/s | 1.2 MB 00:00 2026-02-22T21:28:12.481 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:12.481 INFO:teuthology.orchestra.run.trial051.stdout:Total 9.5 MB/s | 1.2 MB 00:00 2026-02-22T21:28:12.481 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:12.485 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:12.485 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:12.497 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:12.498 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:12.498 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:12.519 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librbd1 2026-02-22T21:28:12.521 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:12.521 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:12.537 INFO:teuthology.orchestra.run.trial068.stdout:ceph-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 9.5 MB/s | 1.2 MB 00:00 2026-02-22T21:28:12.538 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:12.538 INFO:teuthology.orchestra.run.trial068.stdout:Total 9.4 MB/s | 1.2 MB 00:00 2026-02-22T21:28:12.538 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:12.542 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:12.542 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:12.578 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:12.578 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:12.586 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:12.623 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:12.640 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.651 INFO:teuthology.orchestra.run.trial125.stdout:Package python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:12.669 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:12.670 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:12.670 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:12.681 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:12.691 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-cephfs 2026-02-22T21:28:12.698 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.830 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:12.850 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.895 INFO:teuthology.orchestra.run.trial140.stdout:Package librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:12.908 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:12.909 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:12.909 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:12.909 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:12.909 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:12.909 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:12.913 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:12.913 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:12.914 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:12.935 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rados 2026-02-22T21:28:12.937 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-volume 2026-02-22T21:28:12.945 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:13.000 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:13.001 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:13.029 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-volume 2026-02-22T21:28:13.066 INFO:teuthology.orchestra.run.trial125.stdout:Package python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:13.084 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:13.085 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:13.085 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:13.107 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rbd 2026-02-22T21:28:13.244 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:13.246 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:13.310 INFO:teuthology.orchestra.run.trial140.stdout:Package python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:13.312 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:13.328 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:13.329 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:13.329 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:13.330 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:13.331 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:13.331 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:13.340 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:13.351 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rgw 2026-02-22T21:28:13.351 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados-devel 2026-02-22T21:28:13.406 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T21:28:13.420 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:13.425 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:13.425 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:13.426 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:13.448 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados-devel 2026-02-22T21:28:13.485 INFO:teuthology.orchestra.run.trial125.stdout:Package python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:13.503 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:13.504 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:28:13.504 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:13.525 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-fuse 2026-02-22T21:28:13.658 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:13.662 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:13.727 INFO:teuthology.orchestra.run.trial140.stdout:Package python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 126 k 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:13.741 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:13.742 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 126 k 2026-02-22T21:28:13.742 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 456 k 2026-02-22T21:28:13.742 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:13.745 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:13.745 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:13.745 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:13.757 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:42 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:13.769 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-cephfs 2026-02-22T21:28:13.828 INFO:teuthology.orchestra.run.trial051.stdout:librados-devel-19.2.3-689.gf90f4eae.el9.x86_64. 1.5 MB/s | 126 kB 00:00 2026-02-22T21:28:13.829 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:13.829 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-22T21:28:13.829 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:13.831 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:13.831 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:13.836 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:13.841 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:13.841 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:13.841 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 126 k 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:13.842 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:13.843 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:13.843 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:13.843 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 126 k 2026-02-22T21:28:13.843 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 456 k 2026-02-22T21:28:13.843 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:13.866 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:13.867 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout: rbd-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 85 k 2026-02-22T21:28:13.919 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 85 k 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 227 k 2026-02-22T21:28:13.920 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:13.921 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:13.926 INFO:teuthology.orchestra.run.trial068.stdout:librados-devel-19.2.3-689.gf90f4eae.el9.x86_64. 1.4 MB/s | 126 kB 00:00 2026-02-22T21:28:13.927 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:13.927 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-22T21:28:13.927 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:13.930 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:13.930 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:13.938 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:13.962 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:13.962 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:13.994 INFO:teuthology.orchestra.run.trial125.stdout:rbd-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:13.994 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:13.995 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:13.995 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:13.998 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:13.998 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:14.017 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.017 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:14.029 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:14.029 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:14.033 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:14.073 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:14.079 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:14.080 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:14.097 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.100 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install libcephfs2 2026-02-22T21:28:14.115 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.146 INFO:teuthology.orchestra.run.trial140.stdout:Package python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:14.164 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:14.165 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:14.165 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:14.172 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.172 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:14.172 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:14.173 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:14.173 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:14.173 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:14.180 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.187 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rbd 2026-02-22T21:28:14.200 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs2 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:14.240 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:14.269 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-mirror 2026-02-22T21:28:14.425 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:14.491 INFO:teuthology.orchestra.run.trial051.stdout:Package libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:14.502 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:14.509 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:14.510 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:14.510 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:14.518 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:14.532 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install libcephfs-devel 2026-02-22T21:28:14.567 INFO:teuthology.orchestra.run.trial140.stdout:Package python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:14.582 INFO:teuthology.orchestra.run.trial068.stdout:Package libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:14.585 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:14.586 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:14.586 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:28:14.586 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:14.601 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:14.601 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:14.601 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:14.607 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-fuse 2026-02-22T21:28:14.622 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs-devel 2026-02-22T21:28:14.668 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout: rbd-mirror x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.1 M 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 3.1 M 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 13 M 2026-02-22T21:28:14.669 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:14.824 INFO:teuthology.orchestra.run.trial125.stdout:rbd-mirror-19.2.3-689.gf90f4eae.el9.x86_64.rpm 20 MB/s | 3.1 MB 00:00 2026-02-22T21:28:14.825 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:14.825 INFO:teuthology.orchestra.run.trial125.stdout:Total 20 MB/s | 3.1 MB 00:00 2026-02-22T21:28:14.825 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:14.828 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:14.829 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:14.839 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:14.869 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:14.869 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:14.912 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:14.922 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:14.923 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:14.923 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 34 k 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:14.924 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 34 k 2026-02-22T21:28:14.925 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 153 k 2026-02-22T21:28:14.925 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:14.927 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:43 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:14.987 INFO:teuthology.orchestra.run.trial051.stdout:libcephfs-devel-19.2.3-689.gf90f4eae.el9.x86_64 528 kB/s | 34 kB 00:00 2026-02-22T21:28:14.988 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:14.988 INFO:teuthology.orchestra.run.trial051.stdout:Total 522 kB/s | 34 kB 00:00 2026-02-22T21:28:14.988 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:14.989 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:14.989 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:14.992 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:14.992 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:14.994 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:14.995 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:14.995 INFO:teuthology.orchestra.run.trial140.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:14.995 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:14.995 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout: rbd-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 85 k 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 85 k 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 227 k 2026-02-22T21:28:14.996 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 34 k 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 34 k 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 153 k 2026-02-22T21:28:15.010 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:15.013 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:15.029 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.063 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:15.064 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.083 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:15.088 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:15.139 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:15.164 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados2 2026-02-22T21:28:15.166 INFO:teuthology.orchestra.run.trial140.stdout:rbd-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 506 kB/s | 85 kB 00:00 2026-02-22T21:28:15.166 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:15.166 INFO:teuthology.orchestra.run.trial140.stdout:Total 504 kB/s | 85 kB 00:00 2026-02-22T21:28:15.166 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:15.167 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:15.167 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:15.170 INFO:teuthology.orchestra.run.trial068.stdout:libcephfs-devel-19.2.3-689.gf90f4eae.el9.x86_64 211 kB/s | 34 kB 00:00 2026-02-22T21:28:15.171 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:15.171 INFO:teuthology.orchestra.run.trial068.stdout:Total 209 kB/s | 34 kB 00:00 2026-02-22T21:28:15.171 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:15.173 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:15.173 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:15.175 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:15.176 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:15.197 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:15.198 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:15.199 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:15.213 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.250 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:15.266 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.266 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:15.322 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:15.348 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.349 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados2 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:15.406 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:15.408 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:15.434 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-mirror 2026-02-22T21:28:15.437 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-nbd 2026-02-22T21:28:15.477 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:15.542 INFO:teuthology.orchestra.run.trial051.stdout:Package librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:15.560 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:15.561 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:15.561 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:15.581 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librbd1 2026-02-22T21:28:15.657 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:44 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:15.721 INFO:teuthology.orchestra.run.trial068.stdout:Package librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:15.740 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:15.740 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:15.740 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:15.747 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:15.749 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:15.760 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librbd1 2026-02-22T21:28:15.831 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:15.832 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:15.832 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:15.832 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:15.832 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout: rbd-mirror x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.1 M 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:15.833 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:15.834 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 3.1 M 2026-02-22T21:28:15.834 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 13 M 2026-02-22T21:28:15.834 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:15.835 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:15.836 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:15.836 INFO:teuthology.orchestra.run.trial125.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:15.836 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:15.836 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:15.836 INFO:teuthology.orchestra.run.trial125.stdout: rbd-nbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 170 k 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:15.837 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 170 k 2026-02-22T21:28:15.838 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 490 k 2026-02-22T21:28:15.838 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:15.885 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:15.920 INFO:teuthology.orchestra.run.trial125.stdout:rbd-nbd-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:15.920 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:15.920 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:15.921 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:15.924 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:15.924 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:15.949 INFO:teuthology.orchestra.run.trial051.stdout:Package librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:15.955 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:15.955 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:15.968 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:15.968 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:15.969 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:15.982 INFO:teuthology.orchestra.run.trial140.stdout:rbd-mirror-19.2.3-689.gf90f4eae.el9.x86_64.rpm 21 MB/s | 3.1 MB 00:00 2026-02-22T21:28:15.982 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:15.983 INFO:teuthology.orchestra.run.trial140.stdout:Total 21 MB/s | 3.1 MB 00:00 2026-02-22T21:28:15.983 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:15.986 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:15.987 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:15.989 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rados 2026-02-22T21:28:16.010 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:16.027 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:16.027 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:16.028 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.066 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:16.108 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.131 INFO:teuthology.orchestra.run.trial068.stdout:Package librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:16.149 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:16.150 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:16.150 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:16.166 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.166 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:16.166 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:16.167 INFO:teuthology.orchestra.run.trial125.stdout: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:16.167 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:16.167 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:16.172 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rados 2026-02-22T21:28:16.195 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:28:16.227 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:16.229 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:16.252 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:16.298 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:16.365 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:16.383 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:16.384 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:16.384 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:16.405 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rgw 2026-02-22T21:28:16.479 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:16.509 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:16.544 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:16.562 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:16.562 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:16.562 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:16.564 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:16.583 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rgw 2026-02-22T21:28:16.592 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout: ceph-base-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 72 M 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:Installing dependencies: 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout: ceph-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 24 M 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:Installing weak dependencies: 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout: ceph-debugsource x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:16.593 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout:Install 3 Packages 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 107 M 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 654 M 2026-02-22T21:28:16.594 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:16.594 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-nbd 2026-02-22T21:28:16.711 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:16.776 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:16.794 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:16.795 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:16.795 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:16.817 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-cephfs 2026-02-22T21:28:16.893 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:45 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:16.906 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:48 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:16.958 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:16.976 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:16.976 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:16.976 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:16.989 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout: rbd-nbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 170 k 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 170 k 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 490 k 2026-02-22T21:28:16.990 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:16.998 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-cephfs 2026-02-22T21:28:17.077 INFO:teuthology.orchestra.run.trial140.stdout:rbd-nbd-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:17.077 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:17.078 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:17.078 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:17.081 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:17.081 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:17.113 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:17.113 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:17.125 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:48 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:17.168 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:17.185 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:17.189 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:17.208 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:17.208 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:17.208 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:17.230 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rbd 2026-02-22T21:28:17.273 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:17.304 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:17.330 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:17.331 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:17.331 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:17.331 INFO:teuthology.orchestra.run.trial140.stdout: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:17.331 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:17.331 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:17.359 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:28:17.369 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:17.388 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:17.388 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:17.388 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:17.409 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rbd 2026-02-22T21:28:17.542 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:48 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:17.607 INFO:teuthology.orchestra.run.trial051.stdout:Package python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:17.626 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:17.626 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:28:17.626 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:17.648 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-fuse 2026-02-22T21:28:17.665 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:00:49 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:17.721 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:46 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout: ceph-base-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 72 M 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Installing dependencies: 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout: ceph-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 24 M 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Installing weak dependencies: 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout: ceph-debugsource x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:17.750 INFO:teuthology.orchestra.run.trial140.stdout:Install 3 Packages 2026-02-22T21:28:17.751 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:17.751 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 107 M 2026-02-22T21:28:17.751 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 654 M 2026-02-22T21:28:17.751 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:17.785 INFO:teuthology.orchestra.run.trial068.stdout:Package python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:28:17.803 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:17.804 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:28:17.804 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:17.825 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-fuse 2026-02-22T21:28:17.957 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:48 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:18.041 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:18.041 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 85 k 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.042 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 85 k 2026-02-22T21:28:18.043 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 227 k 2026-02-22T21:28:18.043 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:18.116 INFO:teuthology.orchestra.run.trial051.stdout:rbd-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:18.116 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:18.116 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:18.116 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:18.119 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:18.119 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:18.135 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:18.150 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:18.150 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:18.201 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:18.218 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:18.219 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 85 k 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 85 k 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 227 k 2026-02-22T21:28:18.220 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:18.294 INFO:teuthology.orchestra.run.trial068.stdout:rbd-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:18.295 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:18.295 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.1 MB/s | 85 kB 00:00 2026-02-22T21:28:18.295 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:18.298 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:18.298 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:18.302 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.330 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:18.330 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.361 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:18.382 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:18.389 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-mirror 2026-02-22T21:28:18.399 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.481 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.539 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:18.539 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.539 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:18.539 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:18.540 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.540 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:18.565 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-mirror 2026-02-22T21:28:18.707 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:49 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:18.790 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.1 M 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 3.1 M 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 13 M 2026-02-22T21:28:18.791 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:18.890 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:47 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:18.957 INFO:teuthology.orchestra.run.trial051.stdout:rbd-mirror-19.2.3-689.gf90f4eae.el9.x86_64.rpm 19 MB/s | 3.1 MB 00:00 2026-02-22T21:28:18.957 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:18.958 INFO:teuthology.orchestra.run.trial051.stdout:Total 19 MB/s | 3.1 MB 00:00 2026-02-22T21:28:18.958 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:18.961 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:18.962 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.1 M 2026-02-22T21:28:18.974 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 3.1 M 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 13 M 2026-02-22T21:28:18.975 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:19.001 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:19.002 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:19.133 INFO:teuthology.orchestra.run.trial068.stdout:rbd-mirror-19.2.3-689.gf90f4eae.el9.x86_64.rpm 20 MB/s | 3.1 MB 00:00 2026-02-22T21:28:19.133 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:19.133 INFO:teuthology.orchestra.run.trial068.stdout:Total 20 MB/s | 3.1 MB 00:00 2026-02-22T21:28:19.133 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:19.138 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:19.138 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:19.178 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:19.178 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:19.193 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:19.194 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:19.218 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:19.369 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:19.371 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.394 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.394 INFO:teuthology.orchestra.run.trial068.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T21:28:19.394 INFO:teuthology.orchestra.run.trial068.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-22T21:28:19.395 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:19.395 INFO:teuthology.orchestra.run.trial068.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T21:28:19.395 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:19.537 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.537 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:19.538 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:19.538 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:19.538 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:19.538 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:19.566 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-nbd 2026-02-22T21:28:19.578 INFO:teuthology.orchestra.run.trial140.stdout:(1/3): ceph-debugsource-19.2.3-689.gf90f4eae.el 5.9 MB/s | 11 MB 00:01 2026-02-22T21:28:19.690 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:19.690 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:19.690 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:19.691 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:19.691 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:19.691 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:19.718 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-nbd 2026-02-22T21:28:19.880 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:50 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:19.963 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:19.963 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:19.963 INFO:teuthology.orchestra.run.trial051.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:19.963 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 170 k 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 170 k 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 490 k 2026-02-22T21:28:19.964 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:20.034 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:49 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:20.052 INFO:teuthology.orchestra.run.trial051.stdout:rbd-nbd-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:20.052 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:20.052 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:20.052 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:20.056 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:20.057 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:20.089 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:20.089 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout: Package Architecture Version Repository Size 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 170 k 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:20.117 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.118 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 170 k 2026-02-22T21:28:20.118 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 490 k 2026-02-22T21:28:20.118 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:20.146 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:20.162 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.204 INFO:teuthology.orchestra.run.trial068.stdout:rbd-nbd-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:20.204 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:20.204 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.9 MB/s | 170 kB 00:00 2026-02-22T21:28:20.205 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:20.208 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:20.208 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:20.239 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:20.239 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:20.243 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.294 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:20.299 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.300 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:20.300 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:20.300 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:20.300 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:20.300 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:20.311 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.328 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:28:20.389 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.446 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:20.475 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:28:20.641 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:00:51 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:20.725 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:20.726 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:20.726 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:20.726 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:20.726 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 72 M 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:Installing dependencies: 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout: ceph-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 24 M 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:Installing weak dependencies: 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout: ceph-debugsource x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:Install 3 Packages 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:20.727 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 107 M 2026-02-22T21:28:20.728 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 654 M 2026-02-22T21:28:20.728 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:20.786 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:00:49 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:20.869 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:20.869 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.869 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 72 M 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Installing dependencies: 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 24 M 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Installing weak dependencies: 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debugsource x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Install 3 Packages 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 107 M 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 654 M 2026-02-22T21:28:20.870 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:28:21.809 INFO:teuthology.orchestra.run.trial140.stdout:(2/3): ceph-debuginfo-19.2.3-689.gf90f4eae.el9. 6.0 MB/s | 24 MB 00:04 2026-02-22T21:28:21.969 INFO:teuthology.orchestra.run.trial051.stdout:(1/3): ceph-debugsource-19.2.3-689.gf90f4eae.el 8.7 MB/s | 11 MB 00:01 2026-02-22T21:28:23.388 INFO:teuthology.orchestra.run.trial051.stdout:(2/3): ceph-debuginfo-19.2.3-689.gf90f4eae.el9. 9.2 MB/s | 24 MB 00:02 2026-02-22T21:28:23.973 INFO:teuthology.orchestra.run.trial125.stdout:(1/3): ceph-base-debuginfo-19.2.3-689.gf90f4eae 9.7 MB/s | 72 MB 00:07 2026-02-22T21:28:24.097 INFO:teuthology.orchestra.run.trial068.stdout:(1/3): ceph-debugsource-19.2.3-689.gf90f4eae.el 3.4 MB/s | 11 MB 00:03 2026-02-22T21:28:24.155 INFO:teuthology.orchestra.run.trial125.stdout:(2/3): ceph-debugsource-19.2.3-689.gf90f4eae.el 1.4 MB/s | 11 MB 00:07 2026-02-22T21:28:26.081 INFO:teuthology.orchestra.run.trial140.stdout:(3/3): ceph-base-debuginfo-19.2.3-689.gf90f4eae 8.6 MB/s | 72 MB 00:08 2026-02-22T21:28:26.083 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:26.083 INFO:teuthology.orchestra.run.trial140.stdout:Total 13 MB/s | 107 MB 00:08 2026-02-22T21:28:26.083 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:28:26.089 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:28:26.089 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:28:26.134 INFO:teuthology.orchestra.run.trial051.stdout:(3/3): ceph-base-debuginfo-19.2.3-689.gf90f4eae 13 MB/s | 72 MB 00:05 2026-02-22T21:28:26.136 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:26.136 INFO:teuthology.orchestra.run.trial051.stdout:Total 20 MB/s | 107 MB 00:05 2026-02-22T21:28:26.137 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:28:26.142 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:28:26.142 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:28:26.467 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:28:26.467 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:28:26.523 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:28:26.523 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:28:26.546 INFO:teuthology.orchestra.run.trial068.stdout:(2/3): ceph-debuginfo-19.2.3-689.gf90f4eae.el9. 4.3 MB/s | 24 MB 00:05 2026-02-22T21:28:27.189 INFO:teuthology.orchestra.run.trial125.stdout:(3/3): ceph-debuginfo-19.2.3-689.gf90f4eae.el9. 2.3 MB/s | 24 MB 00:10 2026-02-22T21:28:27.191 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:27.191 INFO:teuthology.orchestra.run.trial125.stdout:Total 10 MB/s | 107 MB 00:10 2026-02-22T21:28:27.191 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:28:27.196 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:28:27.197 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:28:27.577 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:28:27.578 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:28:27.836 INFO:teuthology.orchestra.run.trial068.stdout:(3/3): ceph-base-debuginfo-19.2.3-689.gf90f4eae 10 MB/s | 72 MB 00:06 2026-02-22T21:28:27.838 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:28:27.838 INFO:teuthology.orchestra.run.trial068.stdout:Total 15 MB/s | 107 MB 00:06 2026-02-22T21:28:27.839 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:28:27.844 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:28:27.844 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:28:28.224 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:28:28.224 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:28:28.935 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:28:28.965 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:28:30.044 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:28:30.207 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 1/3 2026-02-22T21:28:30.230 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 1/3 2026-02-22T21:28:30.678 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:28:31.315 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 1/3 2026-02-22T21:28:31.945 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 1/3 2026-02-22T21:28:33.898 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:33.901 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:33.917 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:33.920 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:34.445 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:34.445 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/3 2026-02-22T21:28:34.445 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:34.482 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:34.482 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/3 2026-02-22T21:28:34.482 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:34.503 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 3/3 2026-02-22T21:28:34.503 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:34.503 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:28:34.504 INFO:teuthology.orchestra.run.trial051.stdout: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.504 INFO:teuthology.orchestra.run.trial051.stdout: ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.504 INFO:teuthology.orchestra.run.trial051.stdout: ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.504 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:34.504 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:28:34.540 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 3/3 2026-02-22T21:28:34.540 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:34.540 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:28:34.540 INFO:teuthology.orchestra.run.trial140.stdout: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.540 INFO:teuthology.orchestra.run.trial140.stdout: ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.541 INFO:teuthology.orchestra.run.trial140.stdout: ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:34.541 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:34.541 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:28:34.545 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-common-debuginfo 2026-02-22T21:28:34.581 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-common-debuginfo 2026-02-22T21:28:34.862 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:01:05 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:34.890 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:01:06 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:28:34.945 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 474 M 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 474 M 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 2.0 G 2026-02-22T21:28:34.946 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:28:34.973 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:28:34.973 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:34.973 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:28:34.973 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout: ceph-common-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 474 M 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 474 M 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 2.0 G 2026-02-22T21:28:34.974 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:28:34.999 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:35.018 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:35.523 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:35.523 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/3 2026-02-22T21:28:35.523 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 3/3 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:35.582 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:28:35.622 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-common-debuginfo 2026-02-22T21:28:35.639 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:35.659 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:35.929 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:01:06 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:28:36.011 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout: ceph-common-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 474 M 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:28:36.012 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:28:36.013 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 474 M 2026-02-22T21:28:36.013 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 2.0 G 2026-02-22T21:28:36.013 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:28:36.201 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 3/3 2026-02-22T21:28:36.201 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/3 2026-02-22T21:28:36.201 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2/3 2026-02-22T21:28:36.258 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 3/3 2026-02-22T21:28:36.258 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:36.258 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:28:36.258 INFO:teuthology.orchestra.run.trial068.stdout: ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:36.259 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:36.259 INFO:teuthology.orchestra.run.trial068.stdout: ceph-debugsource-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:28:36.259 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:36.259 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:28:36.298 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-common-debuginfo 2026-02-22T21:28:36.608 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:01:05 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 474 M 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:36.691 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 474 M 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.0 G 2026-02-22T21:28:36.692 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:29:06.158 INFO:teuthology.orchestra.run.trial051.stdout:ceph-common-debuginfo-19.2.3-689.gf90f4eae.el9. 15 MB/s | 474 MB 00:31 2026-02-22T21:29:06.159 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:06.159 INFO:teuthology.orchestra.run.trial051.stdout:Total 15 MB/s | 474 MB 00:31 2026-02-22T21:29:06.159 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:29:06.161 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:29:06.161 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:29:06.619 INFO:teuthology.orchestra.run.trial125.stdout:ceph-common-debuginfo-19.2.3-689.gf90f4eae.el9. 15 MB/s | 474 MB 00:30 2026-02-22T21:29:06.619 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:06.619 INFO:teuthology.orchestra.run.trial125.stdout:Total 15 MB/s | 474 MB 00:30 2026-02-22T21:29:06.619 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:29:06.622 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:29:06.622 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:29:06.629 INFO:teuthology.orchestra.run.trial068.stdout:ceph-common-debuginfo-19.2.3-689.gf90f4eae.el9. 16 MB/s | 474 MB 00:29 2026-02-22T21:29:06.630 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:06.631 INFO:teuthology.orchestra.run.trial068.stdout:Total 16 MB/s | 474 MB 00:29 2026-02-22T21:29:06.631 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:29:06.633 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:29:06.633 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:29:07.571 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:29:07.572 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:29:08.030 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:29:08.030 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:29:08.044 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:29:08.044 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:29:17.604 INFO:teuthology.orchestra.run.trial140.stdout:ceph-common-debuginfo-19.2.3-689.gf90f4eae.el9. 11 MB/s | 474 MB 00:42 2026-02-22T21:29:17.605 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:17.605 INFO:teuthology.orchestra.run.trial140.stdout:Total 11 MB/s | 474 MB 00:42 2026-02-22T21:29:17.605 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:29:17.607 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:29:17.608 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:29:19.022 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:29:19.022 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:29:31.063 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:29:31.081 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:32.021 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:29:32.039 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:32.605 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:32.664 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:29:32.675 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:29:32.693 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:32.739 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2026-02-22T21:29:33.044 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:02:04 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:29:33.128 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.5 M 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:======================================================================================= 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 1.5 M 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 7.6 M 2026-02-22T21:29:33.129 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:29:33.252 INFO:teuthology.orchestra.run.trial051.stdout:ceph-immutable-object-cache-debuginfo-19.2.3-68 12 MB/s | 1.5 MB 00:00 2026-02-22T21:29:33.252 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:33.252 INFO:teuthology.orchestra.run.trial051.stdout:Total 12 MB/s | 1.5 MB 00:00 2026-02-22T21:29:33.252 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:29:33.254 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:29:33.254 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:29:33.276 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:29:33.276 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:29:33.388 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:29:33.405 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:33.464 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:33.522 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:33.523 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.523 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:29:33.523 INFO:teuthology.orchestra.run.trial051.stdout: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:33.523 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.523 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:29:33.551 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-radosgw-debuginfo 2026-02-22T21:29:33.570 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:33.629 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:29:33.698 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2026-02-22T21:29:33.857 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:02:04 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:33.941 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 200 M 2026-02-22T21:29:33.942 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 200 M 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 893 M 2026-02-22T21:29:33.943 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:29:34.008 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:02:05 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:34.090 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:======================================================================================= 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:======================================================================================= 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.5 M 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:======================================================================================= 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 1.5 M 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 7.6 M 2026-02-22T21:29:34.091 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:29:34.229 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:34.288 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:29:34.363 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2026-02-22T21:29:34.677 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:02:03 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.5 M 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:======================================================================================= 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:29:34.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:34.761 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 1.5 M 2026-02-22T21:29:34.761 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 7.6 M 2026-02-22T21:29:34.761 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:29:35.012 INFO:teuthology.orchestra.run.trial125.stdout:ceph-immutable-object-cache-debuginfo-19.2.3-68 1.7 MB/s | 1.5 MB 00:00 2026-02-22T21:29:35.012 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:35.012 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.7 MB/s | 1.5 MB 00:00 2026-02-22T21:29:35.012 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:29:35.014 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:29:35.014 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:29:35.036 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:29:35.036 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:29:35.148 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:29:35.165 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.224 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:35.282 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:29:35.307 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-radosgw-debuginfo 2026-02-22T21:29:35.330 INFO:teuthology.orchestra.run.trial068.stdout:ceph-immutable-object-cache-debuginfo-19.2.3-68 2.7 MB/s | 1.5 MB 00:00 2026-02-22T21:29:35.330 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:35.330 INFO:teuthology.orchestra.run.trial068.stdout:Total 2.7 MB/s | 1.5 MB 00:00 2026-02-22T21:29:35.330 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:29:35.332 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:29:35.332 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:29:35.354 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:29:35.354 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:29:35.468 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:29:35.485 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.544 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.602 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:35.602 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:35.602 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:29:35.603 INFO:teuthology.orchestra.run.trial068.stdout: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:35.603 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:35.603 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:29:35.618 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:02:06 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:35.629 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-radosgw-debuginfo 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout: ceph-radosgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 200 M 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 200 M 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 893 M 2026-02-22T21:29:35.701 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:29:35.933 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:02:04 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:29:36.016 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:29:36.016 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:29:36.016 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 200 M 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 200 M 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 893 M 2026-02-22T21:29:36.017 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:29:42.554 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:29:42.573 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:44.084 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:44.141 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:29:44.141 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.141 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:29:44.141 INFO:teuthology.orchestra.run.trial140.stdout: ceph-common-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:44.142 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.142 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:29:44.209 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2026-02-22T21:29:44.515 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:29:44.597 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:29:44.597 INFO:teuthology.orchestra.run.trial140.stdout:======================================================================================= 2026-02-22T21:29:44.597 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:29:44.597 INFO:teuthology.orchestra.run.trial140.stdout:======================================================================================= 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.5 M 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:======================================================================================= 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 1.5 M 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 7.6 M 2026-02-22T21:29:44.598 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:29:44.721 INFO:teuthology.orchestra.run.trial140.stdout:ceph-immutable-object-cache-debuginfo-19.2.3-68 12 MB/s | 1.5 MB 00:00 2026-02-22T21:29:44.722 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:44.722 INFO:teuthology.orchestra.run.trial140.stdout:Total 12 MB/s | 1.5 MB 00:00 2026-02-22T21:29:44.722 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:29:44.724 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:29:44.724 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:29:44.746 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:29:44.747 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:29:44.860 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:29:44.878 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:44.936 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:44.994 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.2.3-689.g 1/1 2026-02-22T21:29:44.994 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.994 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:29:44.994 INFO:teuthology.orchestra.run.trial140.stdout: ceph-immutable-object-cache-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:44.994 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:44.995 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:29:45.020 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-radosgw-debuginfo 2026-02-22T21:29:45.332 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:29:45.415 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout: ceph-radosgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 200 M 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 200 M 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 893 M 2026-02-22T21:29:45.416 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:29:47.521 INFO:teuthology.orchestra.run.trial125.stdout:ceph-radosgw-debuginfo-19.2.3-689.gf90f4eae.el9 17 MB/s | 200 MB 00:11 2026-02-22T21:29:47.521 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:47.522 INFO:teuthology.orchestra.run.trial125.stdout:Total 17 MB/s | 200 MB 00:11 2026-02-22T21:29:47.522 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:29:47.524 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:29:47.524 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:29:47.779 INFO:teuthology.orchestra.run.trial051.stdout:ceph-radosgw-debuginfo-19.2.3-689.gf90f4eae.el9 14 MB/s | 200 MB 00:13 2026-02-22T21:29:47.780 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:47.780 INFO:teuthology.orchestra.run.trial051.stdout:Total 14 MB/s | 200 MB 00:13 2026-02-22T21:29:47.780 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:29:47.782 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:29:47.782 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:29:48.128 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:29:48.129 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:29:48.387 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:29:48.388 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:29:53.446 INFO:teuthology.orchestra.run.trial140.stdout:ceph-radosgw-debuginfo-19.2.3-689.gf90f4eae.el9 25 MB/s | 200 MB 00:08 2026-02-22T21:29:53.447 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:29:53.447 INFO:teuthology.orchestra.run.trial140.stdout:Total 25 MB/s | 200 MB 00:08 2026-02-22T21:29:53.447 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:29:53.449 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:29:53.449 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:29:54.054 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:29:54.054 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:29:58.152 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:29:58.170 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:58.422 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:29:58.440 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:58.860 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:58.918 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:29:58.961 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-test-debuginfo 2026-02-22T21:29:59.135 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:59.193 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:29:59.193 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:59.193 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:29:59.193 INFO:teuthology.orchestra.run.trial051.stdout: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:29:59.193 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:59.194 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:29:59.236 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-test-debuginfo 2026-02-22T21:29:59.270 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:59.351 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout: ceph-test-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 729 M 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 729 M 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 3.6 G 2026-02-22T21:29:59.352 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:29:59.548 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:29:59.631 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 729 M 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:29:59.632 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:29:59.633 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:29:59.633 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 729 M 2026-02-22T21:29:59.633 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 3.6 G 2026-02-22T21:29:59.633 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:30:04.639 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:30:04.657 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:05.303 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:30:05.362 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:30:05.407 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-test-debuginfo 2026-02-22T21:30:05.717 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:30:05.800 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout: ceph-test-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 729 M 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:30:05.801 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:30:05.802 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:30:05.802 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:30:05.802 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 729 M 2026-02-22T21:30:05.802 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 3.6 G 2026-02-22T21:30:05.802 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:30:18.687 INFO:teuthology.orchestra.run.trial068.stdout:ceph-radosgw-debuginfo-19.2.3-689.gf90f4eae.el9 4.7 MB/s | 200 MB 00:42 2026-02-22T21:30:18.688 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:30:18.688 INFO:teuthology.orchestra.run.trial068.stdout:Total 4.7 MB/s | 200 MB 00:42 2026-02-22T21:30:18.688 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:30:18.690 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:30:18.690 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:30:19.298 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:30:19.299 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:30:29.549 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:30:29.566 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:30.265 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x8 1/1 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout: ceph-radosgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:30:30.323 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:30:30.366 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-test-debuginfo 2026-02-22T21:30:30.676 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:02:59 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:30:30.759 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 729 M 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:30:30.760 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:30:30.761 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:30:30.761 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 729 M 2026-02-22T21:30:30.761 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 3.6 G 2026-02-22T21:30:30.761 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:31:03.060 INFO:teuthology.orchestra.run.trial125.stdout:ceph-test-debuginfo-19.2.3-689.gf90f4eae.el9.x8 11 MB/s | 729 MB 01:03 2026-02-22T21:31:03.061 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:03.062 INFO:teuthology.orchestra.run.trial125.stdout:Total 11 MB/s | 729 MB 01:03 2026-02-22T21:31:03.062 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:31:03.064 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:31:03.065 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:31:05.221 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:31:05.222 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:31:15.924 INFO:teuthology.orchestra.run.trial140.stdout:ceph-test-debuginfo-19.2.3-689.gf90f4eae.el9.x8 10 MB/s | 729 MB 01:10 2026-02-22T21:31:15.925 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:15.925 INFO:teuthology.orchestra.run.trial140.stdout:Total 10 MB/s | 729 MB 01:10 2026-02-22T21:31:15.925 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:31:15.928 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:31:15.928 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:31:17.539 INFO:teuthology.orchestra.run.trial051.stdout:ceph-test-debuginfo-19.2.3-689.gf90f4eae.el9.x8 9.4 MB/s | 729 MB 01:17 2026-02-22T21:31:17.539 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:17.539 INFO:teuthology.orchestra.run.trial051.stdout:Total 9.4 MB/s | 729 MB 01:17 2026-02-22T21:31:17.540 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:31:17.543 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:31:17.543 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:31:18.094 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:31:18.094 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:31:19.702 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:31:19.702 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:31:35.331 INFO:teuthology.orchestra.run.trial068.stdout:ceph-test-debuginfo-19.2.3-689.gf90f4eae.el9.x8 11 MB/s | 729 MB 01:04 2026-02-22T21:31:35.332 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:35.332 INFO:teuthology.orchestra.run.trial068.stdout:Total 11 MB/s | 729 MB 01:04 2026-02-22T21:31:35.332 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:31:35.336 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:31:35.336 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:31:37.498 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:31:37.498 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:31:43.040 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:31:43.060 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:43.731 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:43.789 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:43.790 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:43.790 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:31:43.790 INFO:teuthology.orchestra.run.trial125.stdout: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:31:43.790 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:43.790 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:31:43.883 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:31:44.193 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:04:15 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:31:44.257 INFO:teuthology.orchestra.run.trial125.stdout:Package ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:31:44.275 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:31:44.276 INFO:teuthology.orchestra.run.trial125.stdout:Nothing to do. 2026-02-22T21:31:44.276 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:31:44.297 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mgr-debuginfo 2026-02-22T21:31:44.611 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:04:15 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:31:44.693 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:31:44.693 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:44.693 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 18 M 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 18 M 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 110 M 2026-02-22T21:31:44.694 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:31:46.562 INFO:teuthology.orchestra.run.trial125.stdout:ceph-mgr-debuginfo-19.2.3-689.gf90f4eae.el9.x86 9.6 MB/s | 18 MB 00:01 2026-02-22T21:31:46.562 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:46.562 INFO:teuthology.orchestra.run.trial125.stdout:Total 9.6 MB/s | 18 MB 00:01 2026-02-22T21:31:46.562 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:31:46.564 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:31:46.564 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:31:46.635 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:31:46.635 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:31:47.665 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:31:47.683 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:31:47.809 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:31:47.869 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:31:47.869 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:47.869 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:31:47.870 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:31:47.870 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:47.870 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:31:47.897 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mds-debuginfo 2026-02-22T21:31:48.209 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:04:19 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:31:48.291 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mds-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 55 M 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 211 M 2026-02-22T21:31:48.292 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:31:55.830 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:31:55.850 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:56.773 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:31:56.832 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:31:56.925 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:31:57.241 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:04:29 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:31:57.304 INFO:teuthology.orchestra.run.trial140.stdout:Package ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:31:57.323 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:31:57.324 INFO:teuthology.orchestra.run.trial140.stdout:Nothing to do. 2026-02-22T21:31:57.324 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:31:57.345 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mgr-debuginfo 2026-02-22T21:31:57.381 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:31:57.401 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:57.659 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:04:29 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:31:57.742 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:31:57.742 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:31:57.742 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:31:57.742 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:31:57.742 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 18 M 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 18 M 2026-02-22T21:31:57.743 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 110 M 2026-02-22T21:31:57.744 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:31:59.670 INFO:teuthology.orchestra.run.trial140.stdout:ceph-mgr-debuginfo-19.2.3-689.gf90f4eae.el9.x86 9.3 MB/s | 18 MB 00:01 2026-02-22T21:31:59.670 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:31:59.671 INFO:teuthology.orchestra.run.trial140.stdout:Total 9.3 MB/s | 18 MB 00:01 2026-02-22T21:31:59.671 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:31:59.672 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:31:59.672 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:31:59.743 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:31:59.743 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:31:59.913 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:31:59.971 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:32:00.072 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:32:00.386 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:04:31 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:00.451 INFO:teuthology.orchestra.run.trial051.stdout:Package ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:32:00.469 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:32:00.470 INFO:teuthology.orchestra.run.trial051.stdout:Nothing to do. 2026-02-22T21:32:00.470 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:32:00.492 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mgr-debuginfo 2026-02-22T21:32:00.762 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:32:00.780 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:00.799 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:04:31 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:00.883 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:32:00.883 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:00.883 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 18 M 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:00.884 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 18 M 2026-02-22T21:32:00.885 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 110 M 2026-02-22T21:32:00.885 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:32:00.913 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:00.972 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:32:01.001 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mds-debuginfo 2026-02-22T21:32:01.307 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:04:33 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:32:01.391 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mds-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:01.392 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:32:01.393 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:01.393 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 55 M 2026-02-22T21:32:01.393 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 211 M 2026-02-22T21:32:01.393 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:32:02.713 INFO:teuthology.orchestra.run.trial125.stdout:ceph-mds-debuginfo-19.2.3-689.gf90f4eae.el9.x86 3.8 MB/s | 55 MB 00:14 2026-02-22T21:32:02.713 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:02.714 INFO:teuthology.orchestra.run.trial125.stdout:Total 3.8 MB/s | 55 MB 00:14 2026-02-22T21:32:02.714 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:32:02.716 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:32:02.716 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:32:02.896 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:32:02.896 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:32:05.570 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:32:05.588 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:05.786 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:05.844 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:05.845 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:05.845 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:32:05.845 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:05.845 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:05.845 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:32:05.876 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-mon-debuginfo 2026-02-22T21:32:06.188 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:04:37 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mon-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 102 M 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:06.270 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:32:06.271 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:06.271 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 102 M 2026-02-22T21:32:06.271 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 497 M 2026-02-22T21:32:06.271 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:32:06.501 INFO:teuthology.orchestra.run.trial140.stdout:ceph-mds-debuginfo-19.2.3-689.gf90f4eae.el9.x86 11 MB/s | 55 MB 00:05 2026-02-22T21:32:06.502 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:06.503 INFO:teuthology.orchestra.run.trial140.stdout:Total 11 MB/s | 55 MB 00:05 2026-02-22T21:32:06.503 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:32:06.505 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:32:06.505 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:32:06.685 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:32:06.685 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:32:07.820 INFO:teuthology.orchestra.run.trial051.stdout:ceph-mgr-debuginfo-19.2.3-689.gf90f4eae.el9.x86 2.6 MB/s | 18 MB 00:06 2026-02-22T21:32:07.821 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:07.821 INFO:teuthology.orchestra.run.trial051.stdout:Total 2.6 MB/s | 18 MB 00:06 2026-02-22T21:32:07.822 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:32:07.823 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:32:07.823 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:32:07.894 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:32:07.894 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:32:08.912 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:32:08.930 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.054 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.112 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.112 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:09.112 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:32:09.113 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:09.113 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:09.113 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:32:09.142 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mds-debuginfo 2026-02-22T21:32:09.354 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:32:09.372 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.450 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:04:40 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:09.534 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:32:09.534 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:09.535 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 55 M 2026-02-22T21:32:09.536 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 211 M 2026-02-22T21:32:09.536 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:32:09.571 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:09.630 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:32:09.662 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-mon-debuginfo 2026-02-22T21:32:09.971 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:04:41 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mon-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 102 M 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:32:10.052 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:10.053 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 102 M 2026-02-22T21:32:10.053 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 497 M 2026-02-22T21:32:10.053 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:32:12.321 INFO:teuthology.orchestra.run.trial125.stdout:ceph-mon-debuginfo-19.2.3-689.gf90f4eae.el9.x86 17 MB/s | 102 MB 00:06 2026-02-22T21:32:12.322 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:12.322 INFO:teuthology.orchestra.run.trial125.stdout:Total 17 MB/s | 102 MB 00:06 2026-02-22T21:32:12.322 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:32:12.323 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:32:12.324 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:32:12.639 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:32:12.639 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:32:12.729 INFO:teuthology.orchestra.run.trial051.stdout:ceph-mds-debuginfo-19.2.3-689.gf90f4eae.el9.x86 17 MB/s | 55 MB 00:03 2026-02-22T21:32:12.730 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:12.730 INFO:teuthology.orchestra.run.trial051.stdout:Total 17 MB/s | 55 MB 00:03 2026-02-22T21:32:12.731 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:32:12.732 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:32:12.733 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:32:12.913 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:32:12.913 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:32:15.235 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:32:15.256 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:32:15.588 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:32:15.605 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:15.798 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:15.855 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:15.855 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:15.856 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:32:15.856 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:15.856 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:15.856 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:32:15.888 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-mon-debuginfo 2026-02-22T21:32:16.198 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:04:47 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:32:16.281 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 102 M 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 102 M 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 497 M 2026-02-22T21:32:16.282 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:32:17.307 INFO:teuthology.orchestra.run.trial140.stdout:ceph-mon-debuginfo-19.2.3-689.gf90f4eae.el9.x86 14 MB/s | 102 MB 00:07 2026-02-22T21:32:17.308 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:17.308 INFO:teuthology.orchestra.run.trial140.stdout:Total 14 MB/s | 102 MB 00:07 2026-02-22T21:32:17.308 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:32:17.310 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:32:17.310 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:32:17.627 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:32:17.627 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:32:17.809 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:32:17.856 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:32:17.866 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:32:17.866 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:17.866 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:32:17.867 INFO:teuthology.orchestra.run.trial068.stdout: ceph-test-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:17.867 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:17.867 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:32:17.873 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:17.961 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-base-debuginfo 2026-02-22T21:32:18.269 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:04:47 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:32:18.289 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:18.333 INFO:teuthology.orchestra.run.trial068.stdout:Package ceph-base-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:18.348 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:32:18.351 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:32:18.352 INFO:teuthology.orchestra.run.trial068.stdout:Nothing to do. 2026-02-22T21:32:18.352 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:32:18.373 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mgr-debuginfo 2026-02-22T21:32:18.386 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-osd-debuginfo 2026-02-22T21:32:18.685 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:04:47 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:32:18.699 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:04:49 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:18.768 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:32:18.768 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:18.768 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:32:18.768 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:18.768 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 18 M 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 18 M 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 110 M 2026-02-22T21:32:18.769 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:32:18.781 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:32:18.782 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:18.782 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:32:18.782 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:18.782 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout: ceph-osd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 298 M 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 298 M 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 1.4 G 2026-02-22T21:32:18.783 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:32:21.190 INFO:teuthology.orchestra.run.trial068.stdout:ceph-mgr-debuginfo-19.2.3-689.gf90f4eae.el9.x86 7.4 MB/s | 18 MB 00:02 2026-02-22T21:32:21.190 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:21.190 INFO:teuthology.orchestra.run.trial068.stdout:Total 7.4 MB/s | 18 MB 00:02 2026-02-22T21:32:21.191 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:32:21.192 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:32:21.192 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:32:21.262 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:32:21.263 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:32:22.298 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:32:22.315 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:22.446 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:22.504 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:22.504 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:22.504 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:32:22.504 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mgr-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:22.504 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:22.505 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:32:22.533 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mds-debuginfo 2026-02-22T21:32:22.829 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:32:22.842 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:04:51 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:32:22.848 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:22.925 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:32:22.925 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:22.925 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:32:22.925 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 55 M 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 211 M 2026-02-22T21:32:22.926 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:32:23.269 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:23.328 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:32:23.364 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-osd-debuginfo 2026-02-22T21:32:23.677 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:04:55 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:32:23.759 INFO:teuthology.orchestra.run.trial140.stdout: ceph-osd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 298 M 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 298 M 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 1.4 G 2026-02-22T21:32:23.760 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:32:24.148 INFO:teuthology.orchestra.run.trial051.stdout:ceph-mon-debuginfo-19.2.3-689.gf90f4eae.el9.x86 13 MB/s | 102 MB 00:07 2026-02-22T21:32:24.149 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:24.149 INFO:teuthology.orchestra.run.trial051.stdout:Total 13 MB/s | 102 MB 00:07 2026-02-22T21:32:24.149 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:32:24.151 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:32:24.151 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:32:24.467 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:32:24.467 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:32:28.089 INFO:teuthology.orchestra.run.trial068.stdout:ceph-mds-debuginfo-19.2.3-689.gf90f4eae.el9.x86 11 MB/s | 55 MB 00:05 2026-02-22T21:32:28.089 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:28.090 INFO:teuthology.orchestra.run.trial068.stdout:Total 11 MB/s | 55 MB 00:05 2026-02-22T21:32:28.090 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:32:28.091 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:32:28.092 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:32:28.271 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:32:28.271 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:32:29.779 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:32:29.797 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:30.211 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:30.269 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:30.269 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:30.270 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:32:30.270 INFO:teuthology.orchestra.run.trial051.stdout: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:30.270 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:30.270 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:32:30.305 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-osd-debuginfo 2026-02-22T21:32:30.620 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:01 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:32:30.701 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 298 M 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 298 M 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 1.4 G 2026-02-22T21:32:30.702 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:32:30.941 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:32:30.959 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:31.177 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:31.235 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:32:31.236 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:31.236 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:32:31.236 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mds-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:32:31.236 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:31.236 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:32:31.266 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-mon-debuginfo 2026-02-22T21:32:31.580 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:05:00 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:32:31.661 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 102 M 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 102 M 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 497 M 2026-02-22T21:32:31.662 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:32:51.660 INFO:teuthology.orchestra.run.trial125.stdout:ceph-osd-debuginfo-19.2.3-689.gf90f4eae.el9.x86 9.1 MB/s | 298 MB 00:32 2026-02-22T21:32:51.661 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:51.661 INFO:teuthology.orchestra.run.trial125.stdout:Total 9.1 MB/s | 298 MB 00:32 2026-02-22T21:32:51.661 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:32:51.663 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:32:51.663 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:32:52.557 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:32:52.557 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:32:59.205 INFO:teuthology.orchestra.run.trial068.stdout:ceph-mon-debuginfo-19.2.3-689.gf90f4eae.el9.x86 3.7 MB/s | 102 MB 00:27 2026-02-22T21:32:59.206 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:59.206 INFO:teuthology.orchestra.run.trial068.stdout:Total 3.7 MB/s | 102 MB 00:27 2026-02-22T21:32:59.206 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:32:59.207 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:32:59.208 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:32:59.524 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:32:59.524 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:32:59.629 INFO:teuthology.orchestra.run.trial051.stdout:ceph-osd-debuginfo-19.2.3-689.gf90f4eae.el9.x86 10 MB/s | 298 MB 00:28 2026-02-22T21:32:59.629 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:32:59.630 INFO:teuthology.orchestra.run.trial051.stdout:Total 10 MB/s | 298 MB 00:28 2026-02-22T21:32:59.630 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:32:59.632 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:32:59.632 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:00.524 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:00.524 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:04.752 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:04.770 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:05.144 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout: ceph-mon-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:05.202 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:05.237 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-osd-debuginfo 2026-02-22T21:33:05.548 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:05:34 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:05.632 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 298 M 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:05.633 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 298 M 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 1.4 G 2026-02-22T21:33:05.634 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:05.752 INFO:teuthology.orchestra.run.trial140.stdout:ceph-osd-debuginfo-19.2.3-689.gf90f4eae.el9.x86 7.1 MB/s | 298 MB 00:41 2026-02-22T21:33:05.753 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:05.753 INFO:teuthology.orchestra.run.trial140.stdout:Total 7.1 MB/s | 298 MB 00:41 2026-02-22T21:33:05.754 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:05.755 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:05.755 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:06.651 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:06.651 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:07.555 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:07.573 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:08.591 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:08.649 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:08.649 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:08.650 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:08.650 INFO:teuthology.orchestra.run.trial125.stdout: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:08.650 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:08.650 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:08.702 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install ceph-fuse-debuginfo 2026-02-22T21:33:09.010 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:40 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:09.093 INFO:teuthology.orchestra.run.trial125.stdout: ceph-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.8 M 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 6.8 M 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 36 M 2026-02-22T21:33:09.094 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:11.313 INFO:teuthology.orchestra.run.trial125.stdout:ceph-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x8 3.1 MB/s | 6.8 MB 00:02 2026-02-22T21:33:11.314 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:11.314 INFO:teuthology.orchestra.run.trial125.stdout:Total 3.1 MB/s | 6.8 MB 00:02 2026-02-22T21:33:11.315 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:11.316 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:11.316 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:11.354 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:11.354 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:11.754 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:11.771 INFO:teuthology.orchestra.run.trial125.stdout: Installing : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:11.857 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:11.915 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:11.941 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librados-devel-debuginfo 2026-02-22T21:33:12.254 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:43 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:12.338 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:12.338 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout: librados-devel-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 112 k 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:12.339 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 112 k 2026-02-22T21:33:12.340 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 493 k 2026-02-22T21:33:12.340 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:12.782 INFO:teuthology.orchestra.run.trial125.stdout:librados-devel-debuginfo-19.2.3-689.gf90f4eae.e 253 kB/s | 112 kB 00:00 2026-02-22T21:33:12.782 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:12.782 INFO:teuthology.orchestra.run.trial125.stdout:Total 252 kB/s | 112 kB 00:00 2026-02-22T21:33:12.782 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:12.784 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:12.784 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:12.802 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:12.803 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:12.841 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:12.859 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:12.915 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:12.973 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:12.999 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install libcephfs2-debuginfo 2026-02-22T21:33:13.313 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:44 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:13.396 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.0 M 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 6.0 M 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 31 M 2026-02-22T21:33:13.397 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:14.397 INFO:teuthology.orchestra.run.trial125.stdout:libcephfs2-debuginfo-19.2.3-689.gf90f4eae.el9.x 6.0 MB/s | 6.0 MB 00:00 2026-02-22T21:33:14.397 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:14.398 INFO:teuthology.orchestra.run.trial125.stdout:Total 6.0 MB/s | 6.0 MB 00:00 2026-02-22T21:33:14.398 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:14.399 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:14.399 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:14.434 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:14.434 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:14.600 INFO:teuthology.orchestra.run.trial068.stdout:ceph-osd-debuginfo-19.2.3-689.gf90f4eae.el9.x86 33 MB/s | 298 MB 00:08 2026-02-22T21:33:14.601 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:14.601 INFO:teuthology.orchestra.run.trial068.stdout:Total 33 MB/s | 298 MB 00:08 2026-02-22T21:33:14.601 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:14.603 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:14.603 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:14.779 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:14.797 INFO:teuthology.orchestra.run.trial125.stdout: Installing : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:14.882 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:14.941 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:14.941 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:14.941 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:14.941 INFO:teuthology.orchestra.run.trial125.stdout: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:14.942 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:14.942 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:14.969 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librados2-debuginfo 2026-02-22T21:33:15.280 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:46 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout: librados2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 47 M 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:15.362 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 47 M 2026-02-22T21:33:15.363 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 302 M 2026-02-22T21:33:15.363 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:15.495 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:15.495 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:16.045 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:16.063 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:16.335 INFO:teuthology.orchestra.run.trial125.stdout:librados2-debuginfo-19.2.3-689.gf90f4eae.el9.x8 49 MB/s | 47 MB 00:00 2026-02-22T21:33:16.336 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:16.336 INFO:teuthology.orchestra.run.trial125.stdout:Total 48 MB/s | 47 MB 00:00 2026-02-22T21:33:16.337 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:16.338 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:16.339 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:16.496 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:16.496 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:17.099 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:17.157 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:17.212 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install ceph-fuse-debuginfo 2026-02-22T21:33:17.524 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:48 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:17.607 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.8 M 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:17.608 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 6.8 M 2026-02-22T21:33:17.609 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 36 M 2026-02-22T21:33:17.609 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:17.827 INFO:teuthology.orchestra.run.trial051.stdout:ceph-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x8 31 MB/s | 6.8 MB 00:00 2026-02-22T21:33:17.828 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:17.828 INFO:teuthology.orchestra.run.trial051.stdout:Total 31 MB/s | 6.8 MB 00:00 2026-02-22T21:33:17.829 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:17.830 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:17.830 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:17.867 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:17.868 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:18.264 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:18.282 INFO:teuthology.orchestra.run.trial051.stdout: Installing : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:18.365 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:18.423 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:18.423 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:18.423 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:18.423 INFO:teuthology.orchestra.run.trial051.stdout: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:18.424 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:18.424 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:18.451 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados-devel-debuginfo 2026-02-22T21:33:18.763 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:49 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:18.846 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 112 k 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:18.847 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 112 k 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 493 k 2026-02-22T21:33:18.848 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:18.933 INFO:teuthology.orchestra.run.trial051.stdout:librados-devel-debuginfo-19.2.3-689.gf90f4eae.e 1.3 MB/s | 112 kB 00:00 2026-02-22T21:33:18.933 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:18.933 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.3 MB/s | 112 kB 00:00 2026-02-22T21:33:18.934 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:18.935 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:18.935 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:18.953 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:18.953 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:18.992 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:19.009 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:19.064 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:19.122 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:19.148 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install libcephfs2-debuginfo 2026-02-22T21:33:19.191 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:19.210 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:19.456 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:50 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:19.472 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:19.532 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:19.532 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:19.533 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:19.533 INFO:teuthology.orchestra.run.trial125.stdout: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:19.533 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:19.533 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:19.541 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.0 M 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:19.542 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 6.0 M 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 31 M 2026-02-22T21:33:19.543 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:19.566 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install librbd1-debuginfo 2026-02-22T21:33:19.746 INFO:teuthology.orchestra.run.trial051.stdout:libcephfs2-debuginfo-19.2.3-689.gf90f4eae.el9.x 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:19.746 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:19.746 INFO:teuthology.orchestra.run.trial051.stdout:Total 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:19.746 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:19.748 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:19.748 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:19.782 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:19.783 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:19.876 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:50 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:19.959 INFO:teuthology.orchestra.run.trial125.stdout: librbd1-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 55 M 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 374 M 2026-02-22T21:33:19.960 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:20.143 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:20.161 INFO:teuthology.orchestra.run.trial051.stdout: Installing : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:20.238 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:20.296 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:20.323 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librados2-debuginfo 2026-02-22T21:33:20.633 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:51 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout: librados2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 47 M 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:20.716 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 47 M 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 302 M 2026-02-22T21:33:20.717 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:21.077 INFO:teuthology.orchestra.run.trial125.stdout:librbd1-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 49 MB/s | 55 MB 00:01 2026-02-22T21:33:21.078 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:21.078 INFO:teuthology.orchestra.run.trial125.stdout:Total 49 MB/s | 55 MB 00:01 2026-02-22T21:33:21.078 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:21.079 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:21.079 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:21.259 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:21.259 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:21.677 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:21.695 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:21.906 INFO:teuthology.orchestra.run.trial051.stdout:librados2-debuginfo-19.2.3-689.gf90f4eae.el9.x8 40 MB/s | 47 MB 00:01 2026-02-22T21:33:21.907 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:21.907 INFO:teuthology.orchestra.run.trial051.stdout:Total 40 MB/s | 47 MB 00:01 2026-02-22T21:33:21.907 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:21.909 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:21.909 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:22.068 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:22.068 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:22.749 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:22.806 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:22.807 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:22.807 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:22.807 INFO:teuthology.orchestra.run.trial140.stdout: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:22.807 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:22.807 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:22.858 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install ceph-fuse-debuginfo 2026-02-22T21:33:23.169 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:05:55 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:23.250 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout: ceph-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.8 M 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 6.8 M 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 36 M 2026-02-22T21:33:23.251 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:23.483 INFO:teuthology.orchestra.run.trial140.stdout:ceph-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x8 30 MB/s | 6.8 MB 00:00 2026-02-22T21:33:23.483 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:23.484 INFO:teuthology.orchestra.run.trial140.stdout:Total 29 MB/s | 6.8 MB 00:00 2026-02-22T21:33:23.484 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:23.486 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:23.486 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:23.523 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:23.523 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:23.915 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:23.933 INFO:teuthology.orchestra.run.trial140.stdout: Installing : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:24.023 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:24.082 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:24.083 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.083 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:24.083 INFO:teuthology.orchestra.run.trial140.stdout: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:24.083 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.083 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:24.109 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librados-devel-debuginfo 2026-02-22T21:33:24.398 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:24.416 INFO:teuthology.orchestra.run.trial125.stdout: Installing : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:24.419 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:05:56 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout: librados-devel-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 112 k 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:24.505 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:24.506 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:24.506 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.506 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 112 k 2026-02-22T21:33:24.506 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 493 k 2026-02-22T21:33:24.506 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:24.592 INFO:teuthology.orchestra.run.trial140.stdout:librados-devel-debuginfo-19.2.3-689.gf90f4eae.e 1.3 MB/s | 112 kB 00:00 2026-02-22T21:33:24.593 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:24.593 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.2 MB/s | 112 kB 00:00 2026-02-22T21:33:24.593 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:24.594 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:24.594 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:24.612 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:24.612 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:24.651 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:24.668 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:24.724 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:24.740 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:24.755 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:24.773 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:24.783 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:24.783 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.784 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:24.784 INFO:teuthology.orchestra.run.trial140.stdout: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:24.784 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:24.784 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:24.803 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:24.803 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:24.803 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:24.804 INFO:teuthology.orchestra.run.trial125.stdout: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:24.804 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:24.804 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:24.809 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install libcephfs2-debuginfo 2026-02-22T21:33:24.832 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-cephfs-debuginfo 2026-02-22T21:33:25.057 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:25.115 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:25.116 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:25.116 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:25.116 INFO:teuthology.orchestra.run.trial051.stdout: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:25.116 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:25.116 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:25.129 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:05:57 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:25.142 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:56 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:25.147 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install librbd1-debuginfo 2026-02-22T21:33:25.212 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:25.212 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:25.212 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.0 M 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:25.213 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 6.0 M 2026-02-22T21:33:25.214 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 31 M 2026-02-22T21:33:25.214 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:25.223 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:25.224 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.224 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:25.224 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.224 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:25.224 INFO:teuthology.orchestra.run.trial125.stdout: python3-cephfs-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 628 k 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 628 k 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 2.1 M 2026-02-22T21:33:25.225 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:25.341 INFO:teuthology.orchestra.run.trial125.stdout:python3-cephfs-debuginfo-19.2.3-689.gf90f4eae.e 5.3 MB/s | 628 kB 00:00 2026-02-22T21:33:25.341 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:25.342 INFO:teuthology.orchestra.run.trial125.stdout:Total 5.2 MB/s | 628 kB 00:00 2026-02-22T21:33:25.342 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:25.343 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:25.343 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:25.363 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:25.363 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:25.419 INFO:teuthology.orchestra.run.trial140.stdout:libcephfs2-debuginfo-19.2.3-689.gf90f4eae.el9.x 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:25.419 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:25.419 INFO:teuthology.orchestra.run.trial140.stdout:Total 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:25.420 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:25.421 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:25.421 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:25.425 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:25.443 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:25.457 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:25.457 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:25.458 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:05:56 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:25.500 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:25.542 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:25.543 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:25.543 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:25.543 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:25.543 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:25.543 INFO:teuthology.orchestra.run.trial051.stdout: librbd1-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 55 M 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 374 M 2026-02-22T21:33:25.544 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.558 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:25.585 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rados-debuginfo 2026-02-22T21:33:25.803 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:25.821 INFO:teuthology.orchestra.run.trial140.stdout: Installing : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:25.897 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:56 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:25.898 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:25.956 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:25.956 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:25.956 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:25.956 INFO:teuthology.orchestra.run.trial140.stdout: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:25.957 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:25.957 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout: python3-rados-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 779 k 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:25.981 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:25.982 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:25.982 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 779 k 2026-02-22T21:33:25.982 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 2.8 M 2026-02-22T21:33:25.982 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:25.982 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librados2-debuginfo 2026-02-22T21:33:26.292 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:05:58 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:26.375 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout: librados2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 47 M 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:26.376 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:26.377 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:26.377 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 47 M 2026-02-22T21:33:26.377 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 302 M 2026-02-22T21:33:26.377 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:26.607 INFO:teuthology.orchestra.run.trial125.stdout:python3-rados-debuginfo-19.2.3-689.gf90f4eae.el 1.2 MB/s | 779 kB 00:00 2026-02-22T21:33:26.607 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:26.607 INFO:teuthology.orchestra.run.trial125.stdout:Total 1.2 MB/s | 779 kB 00:00 2026-02-22T21:33:26.607 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:26.609 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:26.609 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:26.629 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:26.630 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:26.657 INFO:teuthology.orchestra.run.trial051.stdout:librbd1-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 49 MB/s | 55 MB 00:01 2026-02-22T21:33:26.658 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:26.658 INFO:teuthology.orchestra.run.trial051.stdout:Total 49 MB/s | 55 MB 00:01 2026-02-22T21:33:26.658 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:26.660 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:26.660 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:26.700 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:26.717 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:26.774 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:26.833 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:26.840 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:26.840 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:26.860 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rbd-debuginfo 2026-02-22T21:33:27.167 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:58 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:27.250 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout: python3-rbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:27.251 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 1.1 M 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 4.3 M 2026-02-22T21:33:27.252 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:27.501 INFO:teuthology.orchestra.run.trial140.stdout:librados2-debuginfo-19.2.3-689.gf90f4eae.el9.x8 42 MB/s | 47 MB 00:01 2026-02-22T21:33:27.502 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:27.502 INFO:teuthology.orchestra.run.trial140.stdout:Total 42 MB/s | 47 MB 00:01 2026-02-22T21:33:27.502 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:27.504 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:27.504 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:27.507 INFO:teuthology.orchestra.run.trial125.stdout:python3-rbd-debuginfo-19.2.3-689.gf90f4eae.el9. 4.4 MB/s | 1.1 MB 00:00 2026-02-22T21:33:27.507 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:27.507 INFO:teuthology.orchestra.run.trial125.stdout:Total 4.4 MB/s | 1.1 MB 00:00 2026-02-22T21:33:27.508 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:27.510 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:27.510 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:27.531 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:27.531 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:27.623 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:27.641 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:27.663 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:27.663 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:27.699 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:27.757 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:27.757 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:27.757 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:27.758 INFO:teuthology.orchestra.run.trial125.stdout: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:27.758 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:27.758 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:27.785 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install python3-rgw-debuginfo 2026-02-22T21:33:28.098 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:59 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout: python3-rgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 339 k 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.181 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 339 k 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 1.0 M 2026-02-22T21:33:28.182 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:28.279 INFO:teuthology.orchestra.run.trial125.stdout:python3-rgw-debuginfo-19.2.3-689.gf90f4eae.el9. 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:28.279 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:28.279 INFO:teuthology.orchestra.run.trial125.stdout:Total 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:28.279 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:28.281 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:28.281 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:28.300 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:28.300 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:28.349 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:28.366 INFO:teuthology.orchestra.run.trial125.stdout: Installing : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:28.423 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:28.481 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:28.481 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.482 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:28.482 INFO:teuthology.orchestra.run.trial125.stdout: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:28.482 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.482 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:28.508 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-fuse-debuginfo 2026-02-22T21:33:28.814 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:05:59 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout: rbd-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 547 k 2026-02-22T21:33:28.897 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 547 k 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 3.0 M 2026-02-22T21:33:28.898 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:29.006 INFO:teuthology.orchestra.run.trial125.stdout:rbd-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x86 4.9 MB/s | 547 kB 00:00 2026-02-22T21:33:29.007 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:29.007 INFO:teuthology.orchestra.run.trial125.stdout:Total 4.8 MB/s | 547 kB 00:00 2026-02-22T21:33:29.007 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:29.009 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:29.009 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:29.028 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:29.029 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:29.093 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:29.110 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:29.167 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:29.224 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:29.225 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:29.225 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:29.225 INFO:teuthology.orchestra.run.trial125.stdout: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:29.225 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:29.225 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:29.252 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-mirror-debuginfo 2026-02-22T21:33:29.563 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:06:00 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:29.644 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:29.645 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout: rbd-mirror-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 54 M 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:29.646 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 54 M 2026-02-22T21:33:29.647 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 353 M 2026-02-22T21:33:29.647 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:30.087 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:30.104 INFO:teuthology.orchestra.run.trial051.stdout: Installing : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:30.307 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:30.325 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:30.418 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:30.476 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:30.476 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:30.476 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:30.477 INFO:teuthology.orchestra.run.trial051.stdout: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:30.477 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:30.477 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:30.508 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-cephfs-debuginfo 2026-02-22T21:33:30.602 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:30.639 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:30.656 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:30.660 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:30.660 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:30.661 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:30.661 INFO:teuthology.orchestra.run.trial140.stdout: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:30.661 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:30.661 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:30.691 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install librbd1-debuginfo 2026-02-22T21:33:30.739 INFO:teuthology.orchestra.run.trial125.stdout:rbd-mirror-debuginfo-19.2.3-689.gf90f4eae.el9.x 49 MB/s | 54 MB 00:01 2026-02-22T21:33:30.740 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:30.740 INFO:teuthology.orchestra.run.trial125.stdout:Total 49 MB/s | 54 MB 00:01 2026-02-22T21:33:30.740 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:30.741 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:30.741 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:30.819 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:01 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:30.901 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:30.902 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:30.902 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:30.902 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:30.902 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 628 k 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 628 k 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 2.1 M 2026-02-22T21:33:30.903 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:30.917 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:30.917 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:30.999 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:02 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:31.012 INFO:teuthology.orchestra.run.trial051.stdout:python3-cephfs-debuginfo-19.2.3-689.gf90f4eae.e 5.6 MB/s | 628 kB 00:00 2026-02-22T21:33:31.013 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:31.013 INFO:teuthology.orchestra.run.trial051.stdout:Total 5.6 MB/s | 628 kB 00:00 2026-02-22T21:33:31.013 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:31.014 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:31.014 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:31.033 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:31.034 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:31.081 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout: librbd1-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:31.082 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:31.083 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:31.083 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 55 M 2026-02-22T21:33:31.083 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 374 M 2026-02-22T21:33:31.083 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:31.097 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:31.114 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:31.172 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:31.229 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:31.257 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rados-debuginfo 2026-02-22T21:33:31.570 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:02 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 779 k 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:31.653 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:31.654 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:31.654 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:31.654 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 779 k 2026-02-22T21:33:31.654 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 2.8 M 2026-02-22T21:33:31.654 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:31.733 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:31.791 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:31.791 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:31.791 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:31.791 INFO:teuthology.orchestra.run.trial068.stdout: ceph-osd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:31.792 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:31.792 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:31.846 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install ceph-fuse-debuginfo 2026-02-22T21:33:32.126 INFO:teuthology.orchestra.run.trial051.stdout:python3-rados-debuginfo-19.2.3-689.gf90f4eae.el 1.6 MB/s | 779 kB 00:00 2026-02-22T21:33:32.126 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:32.127 INFO:teuthology.orchestra.run.trial051.stdout:Total 1.6 MB/s | 779 kB 00:00 2026-02-22T21:33:32.127 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:32.128 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:32.128 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:32.147 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:32.148 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:32.163 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:01 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:32.197 INFO:teuthology.orchestra.run.trial140.stdout:librbd1-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 49 MB/s | 55 MB 00:01 2026-02-22T21:33:32.197 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:32.198 INFO:teuthology.orchestra.run.trial140.stdout:Total 49 MB/s | 55 MB 00:01 2026-02-22T21:33:32.198 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:32.200 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:32.200 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:32.222 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:32.239 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.8 M 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:32.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:32.247 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 6.8 M 2026-02-22T21:33:32.247 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 36 M 2026-02-22T21:33:32.247 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:32.296 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:32.354 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:32.380 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rbd-debuginfo 2026-02-22T21:33:32.380 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:32.381 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:32.467 INFO:teuthology.orchestra.run.trial068.stdout:ceph-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x8 31 MB/s | 6.8 MB 00:00 2026-02-22T21:33:32.468 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:32.468 INFO:teuthology.orchestra.run.trial068.stdout:Total 31 MB/s | 6.8 MB 00:00 2026-02-22T21:33:32.468 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:32.470 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:32.470 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:32.507 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:32.508 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:32.689 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:03 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:32.771 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:32.771 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:32.771 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:32.771 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 1.1 M 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 4.3 M 2026-02-22T21:33:32.772 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:32.894 INFO:teuthology.orchestra.run.trial051.stdout:python3-rbd-debuginfo-19.2.3-689.gf90f4eae.el9. 9.4 MB/s | 1.1 MB 00:00 2026-02-22T21:33:32.894 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:32.894 INFO:teuthology.orchestra.run.trial051.stdout:Total 9.3 MB/s | 1.1 MB 00:00 2026-02-22T21:33:32.894 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:32.896 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:32.896 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:32.900 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:32.916 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:32.917 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:32.917 INFO:teuthology.orchestra.run.trial068.stdout: Installing : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:32.997 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:33.005 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:33.023 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout: ceph-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.055 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:33.080 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.082 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados-devel-debuginfo 2026-02-22T21:33:33.139 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.140 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.140 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:33.140 INFO:teuthology.orchestra.run.trial051.stdout: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:33.140 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.140 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:33.168 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install python3-rgw-debuginfo 2026-02-22T21:33:33.388 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:02 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 112 k 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 112 k 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 493 k 2026-02-22T21:33:33.470 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:33.484 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:04 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:33.567 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:33.567 INFO:teuthology.orchestra.run.trial068.stdout:librados-devel-debuginfo-19.2.3-689.gf90f4eae.e 1.1 MB/s | 112 kB 00:00 2026-02-22T21:33:33.567 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 339 k 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 339 k 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 1.0 M 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial068.stdout:Total 1.1 MB/s | 112 kB 00:00 2026-02-22T21:33:33.568 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:33.569 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:33.569 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:33.587 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:33.587 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:33.626 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:33.643 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:33.665 INFO:teuthology.orchestra.run.trial051.stdout:python3-rgw-debuginfo-19.2.3-689.gf90f4eae.el9. 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:33.665 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:33.666 INFO:teuthology.orchestra.run.trial051.stdout:Total 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:33.666 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:33.667 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:33.667 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:33.686 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:33.686 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:33.698 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:33.735 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:33.752 INFO:teuthology.orchestra.run.trial051.stdout: Installing : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout: librados-devel-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:33.756 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:33.782 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install libcephfs2-debuginfo 2026-02-22T21:33:33.809 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.866 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:33.867 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.867 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:33.867 INFO:teuthology.orchestra.run.trial051.stdout: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:33.867 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:33.867 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:33.894 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-fuse-debuginfo 2026-02-22T21:33:33.972 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:33.990 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.094 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:03 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:34.176 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.0 M 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 6.0 M 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 31 M 2026-02-22T21:33:34.177 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:34.207 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:05 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:34.290 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:34.290 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 547 k 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 547 k 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 3.0 M 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:34.291 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:34.352 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:34.382 INFO:teuthology.orchestra.run.trial068.stdout:libcephfs2-debuginfo-19.2.3-689.gf90f4eae.el9.x 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:34.382 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:34.382 INFO:teuthology.orchestra.run.trial068.stdout:Total 29 MB/s | 6.0 MB 00:00 2026-02-22T21:33:34.382 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:34.382 DEBUG:teuthology.orchestra.run.trial125:> sudo yum -y install rbd-nbd-debuginfo 2026-02-22T21:33:34.384 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:34.384 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:34.419 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:34.419 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:34.448 INFO:teuthology.orchestra.run.trial051.stdout:rbd-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x86 3.4 MB/s | 547 kB 00:00 2026-02-22T21:33:34.449 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:34.449 INFO:teuthology.orchestra.run.trial051.stdout:Total 3.4 MB/s | 547 kB 00:00 2026-02-22T21:33:34.449 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:34.451 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:34.451 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:34.471 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:34.471 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:34.534 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:34.551 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:34.612 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:34.671 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:34.687 INFO:teuthology.orchestra.run.trial125.stdout:Last metadata expiration check: 0:06:05 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:34.698 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-mirror-debuginfo 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout:Dependencies resolved. 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout: Package Arch Version Repo Size 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout:Installing: 2026-02-22T21:33:34.770 INFO:teuthology.orchestra.run.trial125.stdout: rbd-nbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.4 M 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:Transaction Summary 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:================================================================================ 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:Install 1 Package 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:Total download size: 1.4 M 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:Installed size: 6.6 M 2026-02-22T21:33:34.771 INFO:teuthology.orchestra.run.trial125.stdout:Downloading Packages: 2026-02-22T21:33:34.785 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:34.802 INFO:teuthology.orchestra.run.trial068.stdout: Installing : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.877 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.934 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:34.935 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:34.935 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:34.935 INFO:teuthology.orchestra.run.trial068.stdout: libcephfs2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:34.935 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:34.935 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:34.961 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librados2-debuginfo 2026-02-22T21:33:35.013 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:06 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:35.096 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 54 M 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 54 M 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 353 M 2026-02-22T21:33:35.097 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:35.273 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:04 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:35.313 INFO:teuthology.orchestra.run.trial125.stdout:rbd-nbd-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 2.5 MB/s | 1.4 MB 00:00 2026-02-22T21:33:35.313 INFO:teuthology.orchestra.run.trial125.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:35.313 INFO:teuthology.orchestra.run.trial125.stdout:Total 2.5 MB/s | 1.4 MB 00:00 2026-02-22T21:33:35.314 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction check 2026-02-22T21:33:35.315 INFO:teuthology.orchestra.run.trial125.stdout:Transaction check succeeded. 2026-02-22T21:33:35.315 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction test 2026-02-22T21:33:35.338 INFO:teuthology.orchestra.run.trial125.stdout:Transaction test succeeded. 2026-02-22T21:33:35.338 INFO:teuthology.orchestra.run.trial125.stdout:Running transaction 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:35.357 INFO:teuthology.orchestra.run.trial068.stdout: librados2-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 47 M 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 47 M 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 302 M 2026-02-22T21:33:35.358 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:35.441 INFO:teuthology.orchestra.run.trial125.stdout: Preparing : 1/1 2026-02-22T21:33:35.459 INFO:teuthology.orchestra.run.trial125.stdout: Installing : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.518 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:35.518 INFO:teuthology.orchestra.run.trial125.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.536 INFO:teuthology.orchestra.run.trial140.stdout: Installing : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.576 INFO:teuthology.orchestra.run.trial125.stdout: Verifying : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.576 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:35.576 INFO:teuthology.orchestra.run.trial125.stdout:Installed: 2026-02-22T21:33:35.576 INFO:teuthology.orchestra.run.trial125.stdout: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:35.577 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:33:35.577 INFO:teuthology.orchestra.run.trial125.stdout:Complete! 2026-02-22T21:33:35.604 DEBUG:teuthology.parallel:result is None 2026-02-22T21:33:35.845 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:35.904 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:35.934 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-cephfs-debuginfo 2026-02-22T21:33:36.246 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:08 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:36.327 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout: python3-cephfs-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 628 k 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 628 k 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 2.1 M 2026-02-22T21:33:36.328 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:36.898 INFO:teuthology.orchestra.run.trial140.stdout:python3-cephfs-debuginfo-19.2.3-689.gf90f4eae.e 1.1 MB/s | 628 kB 00:00 2026-02-22T21:33:36.898 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:36.898 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.1 MB/s | 628 kB 00:00 2026-02-22T21:33:36.898 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:36.900 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:36.900 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:36.920 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:36.920 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:36.984 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:37.001 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:37.057 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:37.115 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:37.115 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:37.115 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:37.115 INFO:teuthology.orchestra.run.trial140.stdout: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:37.116 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:37.116 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:37.141 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rados-debuginfo 2026-02-22T21:33:37.448 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:09 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:37.530 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout: python3-rados-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 779 k 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 779 k 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 2.8 M 2026-02-22T21:33:37.531 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:38.075 INFO:teuthology.orchestra.run.trial140.stdout:python3-rados-debuginfo-19.2.3-689.gf90f4eae.el 1.4 MB/s | 779 kB 00:00 2026-02-22T21:33:38.075 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:38.076 INFO:teuthology.orchestra.run.trial140.stdout:Total 1.4 MB/s | 779 kB 00:00 2026-02-22T21:33:38.076 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:38.077 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:38.077 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:38.097 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:38.097 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:38.150 INFO:teuthology.orchestra.run.trial051.stdout:rbd-mirror-debuginfo-19.2.3-689.gf90f4eae.el9.x 18 MB/s | 54 MB 00:03 2026-02-22T21:33:38.151 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:38.151 INFO:teuthology.orchestra.run.trial051.stdout:Total 18 MB/s | 54 MB 00:03 2026-02-22T21:33:38.151 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:38.153 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:38.153 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:38.169 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:38.186 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:38.245 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:38.303 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:38.304 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:38.304 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:38.304 INFO:teuthology.orchestra.run.trial140.stdout: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:38.304 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:38.304 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:38.328 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:38.329 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:38.329 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rbd-debuginfo 2026-02-22T21:33:38.641 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:10 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout: python3-rbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:38.723 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:38.724 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:38.724 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:38.724 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 1.1 M 2026-02-22T21:33:38.724 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 4.3 M 2026-02-22T21:33:38.724 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:38.845 INFO:teuthology.orchestra.run.trial140.stdout:python3-rbd-debuginfo-19.2.3-689.gf90f4eae.el9. 9.4 MB/s | 1.1 MB 00:00 2026-02-22T21:33:38.845 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:38.846 INFO:teuthology.orchestra.run.trial140.stdout:Total 9.3 MB/s | 1.1 MB 00:00 2026-02-22T21:33:38.846 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:38.847 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:38.847 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:38.869 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:38.869 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:38.959 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:38.976 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.033 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.091 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:39.118 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install python3-rgw-debuginfo 2026-02-22T21:33:39.423 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:11 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:39.505 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout: python3-rgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 339 k 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 339 k 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 1.0 M 2026-02-22T21:33:39.506 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:39.603 INFO:teuthology.orchestra.run.trial140.stdout:python3-rgw-debuginfo-19.2.3-689.gf90f4eae.el9. 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:39.604 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:39.604 INFO:teuthology.orchestra.run.trial140.stdout:Total 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:39.604 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:39.606 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:39.606 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:39.625 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:39.626 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:39.676 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:39.694 INFO:teuthology.orchestra.run.trial140.stdout: Installing : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.750 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.808 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:39.809 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.809 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:39.809 INFO:teuthology.orchestra.run.trial140.stdout: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:39.809 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:39.809 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:39.835 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-fuse-debuginfo 2026-02-22T21:33:40.148 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:12 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:40.229 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout: rbd-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 547 k 2026-02-22T21:33:40.230 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 547 k 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 3.0 M 2026-02-22T21:33:40.231 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:40.346 INFO:teuthology.orchestra.run.trial140.stdout:rbd-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x86 4.6 MB/s | 547 kB 00:00 2026-02-22T21:33:40.346 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:40.347 INFO:teuthology.orchestra.run.trial140.stdout:Total 4.6 MB/s | 547 kB 00:00 2026-02-22T21:33:40.347 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:40.348 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:40.348 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:40.367 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:40.368 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:40.431 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:40.448 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:40.504 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.564 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:40.591 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-mirror-debuginfo 2026-02-22T21:33:40.902 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:12 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout: rbd-mirror-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 54 M 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:40.984 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:40.985 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 54 M 2026-02-22T21:33:40.985 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 353 M 2026-02-22T21:33:40.985 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:41.383 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:41.401 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:41.716 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:41.775 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:41.809 DEBUG:teuthology.orchestra.run.trial051:> sudo yum -y install rbd-nbd-debuginfo 2026-02-22T21:33:41.920 INFO:teuthology.orchestra.run.trial068.stdout:librados2-debuginfo-19.2.3-689.gf90f4eae.el9.x8 7.2 MB/s | 47 MB 00:06 2026-02-22T21:33:41.920 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:41.920 INFO:teuthology.orchestra.run.trial068.stdout:Total 7.2 MB/s | 47 MB 00:06 2026-02-22T21:33:41.921 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:41.923 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:41.923 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:42.080 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:42.080 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:42.125 INFO:teuthology.orchestra.run.trial051.stdout:Last metadata expiration check: 0:06:13 ago on Sun 22 Feb 2026 09:27:29 PM UTC. 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:Dependencies resolved. 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout: Package Arch Version Repo Size 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:Installing: 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.4 M 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:Transaction Summary 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:================================================================================ 2026-02-22T21:33:42.209 INFO:teuthology.orchestra.run.trial051.stdout:Install 1 Package 2026-02-22T21:33:42.210 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:42.210 INFO:teuthology.orchestra.run.trial051.stdout:Total download size: 1.4 M 2026-02-22T21:33:42.210 INFO:teuthology.orchestra.run.trial051.stdout:Installed size: 6.6 M 2026-02-22T21:33:42.210 INFO:teuthology.orchestra.run.trial051.stdout:Downloading Packages: 2026-02-22T21:33:42.576 INFO:teuthology.orchestra.run.trial051.stdout:rbd-nbd-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2026-02-22T21:33:42.577 INFO:teuthology.orchestra.run.trial051.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:42.577 INFO:teuthology.orchestra.run.trial051.stdout:Total 3.7 MB/s | 1.4 MB 00:00 2026-02-22T21:33:42.577 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction check 2026-02-22T21:33:42.578 INFO:teuthology.orchestra.run.trial051.stdout:Transaction check succeeded. 2026-02-22T21:33:42.578 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction test 2026-02-22T21:33:42.601 INFO:teuthology.orchestra.run.trial051.stdout:Transaction test succeeded. 2026-02-22T21:33:42.601 INFO:teuthology.orchestra.run.trial051.stdout:Running transaction 2026-02-22T21:33:42.705 INFO:teuthology.orchestra.run.trial051.stdout: Preparing : 1/1 2026-02-22T21:33:42.723 INFO:teuthology.orchestra.run.trial051.stdout: Installing : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:42.782 INFO:teuthology.orchestra.run.trial051.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:42.839 INFO:teuthology.orchestra.run.trial051.stdout: Verifying : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:42.840 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:42.840 INFO:teuthology.orchestra.run.trial051.stdout:Installed: 2026-02-22T21:33:42.840 INFO:teuthology.orchestra.run.trial051.stdout: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:42.840 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:33:42.840 INFO:teuthology.orchestra.run.trial051.stdout:Complete! 2026-02-22T21:33:42.868 DEBUG:teuthology.parallel:result is None 2026-02-22T21:33:43.247 INFO:teuthology.orchestra.run.trial140.stdout:rbd-mirror-debuginfo-19.2.3-689.gf90f4eae.el9.x 24 MB/s | 54 MB 00:02 2026-02-22T21:33:43.248 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:43.248 INFO:teuthology.orchestra.run.trial140.stdout:Total 24 MB/s | 54 MB 00:02 2026-02-22T21:33:43.249 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:43.250 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:43.250 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:43.425 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:43.425 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:44.767 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:44.786 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:45.042 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_6 1/1 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout: librados2-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:45.100 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:45.132 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install librbd1-debuginfo 2026-02-22T21:33:45.445 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:14 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:45.527 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout: librbd1-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 55 M 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 55 M 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 374 M 2026-02-22T21:33:45.528 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:46.597 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:46.614 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:46.922 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:46.982 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:47.012 DEBUG:teuthology.orchestra.run.trial140:> sudo yum -y install rbd-nbd-debuginfo 2026-02-22T21:33:47.322 INFO:teuthology.orchestra.run.trial140.stdout:Last metadata expiration check: 0:06:19 ago on Sun 22 Feb 2026 09:27:28 PM UTC. 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout:Dependencies resolved. 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout: Package Arch Version Repo Size 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout:Installing: 2026-02-22T21:33:47.404 INFO:teuthology.orchestra.run.trial140.stdout: rbd-nbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.4 M 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:Transaction Summary 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:================================================================================ 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:Install 1 Package 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:Total download size: 1.4 M 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:Installed size: 6.6 M 2026-02-22T21:33:47.405 INFO:teuthology.orchestra.run.trial140.stdout:Downloading Packages: 2026-02-22T21:33:47.911 INFO:teuthology.orchestra.run.trial068.stdout:librbd1-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 23 MB/s | 55 MB 00:02 2026-02-22T21:33:47.912 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:47.912 INFO:teuthology.orchestra.run.trial068.stdout:Total 23 MB/s | 55 MB 00:02 2026-02-22T21:33:47.913 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:47.914 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:47.915 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:47.919 INFO:teuthology.orchestra.run.trial140.stdout:rbd-nbd-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 2.6 MB/s | 1.4 MB 00:00 2026-02-22T21:33:47.920 INFO:teuthology.orchestra.run.trial140.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:47.920 INFO:teuthology.orchestra.run.trial140.stdout:Total 2.6 MB/s | 1.4 MB 00:00 2026-02-22T21:33:47.920 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction check 2026-02-22T21:33:47.922 INFO:teuthology.orchestra.run.trial140.stdout:Transaction check succeeded. 2026-02-22T21:33:47.922 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction test 2026-02-22T21:33:47.946 INFO:teuthology.orchestra.run.trial140.stdout:Transaction test succeeded. 2026-02-22T21:33:47.946 INFO:teuthology.orchestra.run.trial140.stdout:Running transaction 2026-02-22T21:33:48.051 INFO:teuthology.orchestra.run.trial140.stdout: Preparing : 1/1 2026-02-22T21:33:48.068 INFO:teuthology.orchestra.run.trial140.stdout: Installing : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:48.095 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:48.095 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:48.127 INFO:teuthology.orchestra.run.trial140.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout: Verifying : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout:Installed: 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:33:48.186 INFO:teuthology.orchestra.run.trial140.stdout:Complete! 2026-02-22T21:33:48.214 DEBUG:teuthology.parallel:result is None 2026-02-22T21:33:51.289 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:51.307 INFO:teuthology.orchestra.run.trial068.stdout: Installing : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:51.646 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:51.703 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:51.703 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:51.703 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:51.703 INFO:teuthology.orchestra.run.trial068.stdout: librbd1-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:51.704 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:51.704 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:51.735 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-cephfs-debuginfo 2026-02-22T21:33:52.040 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:21 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:52.121 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 628 k 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 628 k 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.1 M 2026-02-22T21:33:52.122 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:52.275 INFO:teuthology.orchestra.run.trial068.stdout:python3-cephfs-debuginfo-19.2.3-689.gf90f4eae.e 4.0 MB/s | 628 kB 00:00 2026-02-22T21:33:52.275 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:52.275 INFO:teuthology.orchestra.run.trial068.stdout:Total 4.0 MB/s | 628 kB 00:00 2026-02-22T21:33:52.275 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:52.277 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:52.277 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:52.296 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:52.296 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:52.360 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:52.377 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:52.433 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:52.491 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9. 1/1 2026-02-22T21:33:52.491 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.492 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:52.492 INFO:teuthology.orchestra.run.trial068.stdout: python3-cephfs-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:52.492 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.492 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:52.518 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rados-debuginfo 2026-02-22T21:33:52.823 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:21 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:52.905 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 779 k 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 779 k 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 2.8 M 2026-02-22T21:33:52.906 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:53.015 INFO:teuthology.orchestra.run.trial068.stdout:python3-rados-debuginfo-19.2.3-689.gf90f4eae.el 7.0 MB/s | 779 kB 00:00 2026-02-22T21:33:53.016 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:53.016 INFO:teuthology.orchestra.run.trial068.stdout:Total 6.9 MB/s | 779 kB 00:00 2026-02-22T21:33:53.016 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:53.017 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:53.018 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:53.038 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:53.038 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:53.113 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:53.130 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:53.187 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:53.245 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x 1/1 2026-02-22T21:33:53.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:53.246 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:53.246 INFO:teuthology.orchestra.run.trial068.stdout: python3-rados-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:53.246 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:53.246 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:53.272 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rbd-debuginfo 2026-02-22T21:33:53.583 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:22 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T21:33:53.666 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 1.1 M 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 4.3 M 2026-02-22T21:33:53.667 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:53.786 INFO:teuthology.orchestra.run.trial068.stdout:python3-rbd-debuginfo-19.2.3-689.gf90f4eae.el9. 9.5 MB/s | 1.1 MB 00:00 2026-02-22T21:33:53.787 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:53.787 INFO:teuthology.orchestra.run.trial068.stdout:Total 9.4 MB/s | 1.1 MB 00:00 2026-02-22T21:33:53.787 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:53.789 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:53.789 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:53.810 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:53.810 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:53.900 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:53.917 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:53.974 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:54.031 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:54.031 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.031 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:54.031 INFO:teuthology.orchestra.run.trial068.stdout: python3-rbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:54.032 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.032 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:54.059 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install python3-rgw-debuginfo 2026-02-22T21:33:54.368 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:23 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:54.451 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:54.451 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 339 k 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.452 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 339 k 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 1.0 M 2026-02-22T21:33:54.453 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:54.551 INFO:teuthology.orchestra.run.trial068.stdout:python3-rgw-debuginfo-19.2.3-689.gf90f4eae.el9. 3.4 MB/s | 339 kB 00:00 2026-02-22T21:33:54.551 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:54.551 INFO:teuthology.orchestra.run.trial068.stdout:Total 3.3 MB/s | 339 kB 00:00 2026-02-22T21:33:54.551 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:54.552 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:54.553 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:54.571 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:54.571 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:54.621 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:54.639 INFO:teuthology.orchestra.run.trial068.stdout: Installing : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:54.695 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:54.752 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86 1/1 2026-02-22T21:33:54.753 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.753 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:54.753 INFO:teuthology.orchestra.run.trial068.stdout: python3-rgw-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:54.753 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:54.753 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:54.779 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-fuse-debuginfo 2026-02-22T21:33:55.090 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:24 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:55.173 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 547 k 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.174 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 547 k 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 3.0 M 2026-02-22T21:33:55.175 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:55.283 INFO:teuthology.orchestra.run.trial068.stdout:rbd-fuse-debuginfo-19.2.3-689.gf90f4eae.el9.x86 4.9 MB/s | 547 kB 00:00 2026-02-22T21:33:55.283 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:55.283 INFO:teuthology.orchestra.run.trial068.stdout:Total 4.8 MB/s | 547 kB 00:00 2026-02-22T21:33:55.283 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:55.285 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:55.285 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:55.305 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:55.305 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:33:55.366 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:33:55.383 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:55.443 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout: rbd-fuse-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.502 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:33:55.529 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-mirror-debuginfo 2026-02-22T21:33:55.843 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:24 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:33:55.926 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:33:55.926 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.926 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:33:55.926 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.926 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 54 M 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 54 M 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 353 M 2026-02-22T21:33:55.927 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:33:57.070 INFO:teuthology.orchestra.run.trial068.stdout:rbd-mirror-debuginfo-19.2.3-689.gf90f4eae.el9.x 47 MB/s | 54 MB 00:01 2026-02-22T21:33:57.071 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:33:57.072 INFO:teuthology.orchestra.run.trial068.stdout:Total 47 MB/s | 54 MB 00:01 2026-02-22T21:33:57.072 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:33:57.074 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:33:57.074 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:33:57.249 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:33:57.249 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:34:00.342 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:34:00.360 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:34:00.661 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_ 1/1 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout: rbd-mirror-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:00.719 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:34:00.752 DEBUG:teuthology.orchestra.run.trial068:> sudo yum -y install rbd-nbd-debuginfo 2026-02-22T21:34:01.063 INFO:teuthology.orchestra.run.trial068.stdout:Last metadata expiration check: 0:06:30 ago on Sun 22 Feb 2026 09:27:31 PM UTC. 2026-02-22T21:34:01.146 INFO:teuthology.orchestra.run.trial068.stdout:Dependencies resolved. 2026-02-22T21:34:01.146 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout: Package Arch Version Repo Size 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Installing: 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd-debuginfo x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.4 M 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Transaction Summary 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:================================================================================ 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Install 1 Package 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Total download size: 1.4 M 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Installed size: 6.6 M 2026-02-22T21:34:01.147 INFO:teuthology.orchestra.run.trial068.stdout:Downloading Packages: 2026-02-22T21:34:01.277 INFO:teuthology.orchestra.run.trial068.stdout:rbd-nbd-debuginfo-19.2.3-689.gf90f4eae.el9.x86_ 10 MB/s | 1.4 MB 00:00 2026-02-22T21:34:01.278 INFO:teuthology.orchestra.run.trial068.stdout:-------------------------------------------------------------------------------- 2026-02-22T21:34:01.278 INFO:teuthology.orchestra.run.trial068.stdout:Total 10 MB/s | 1.4 MB 00:00 2026-02-22T21:34:01.278 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction check 2026-02-22T21:34:01.280 INFO:teuthology.orchestra.run.trial068.stdout:Transaction check succeeded. 2026-02-22T21:34:01.280 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction test 2026-02-22T21:34:01.303 INFO:teuthology.orchestra.run.trial068.stdout:Transaction test succeeded. 2026-02-22T21:34:01.303 INFO:teuthology.orchestra.run.trial068.stdout:Running transaction 2026-02-22T21:34:01.407 INFO:teuthology.orchestra.run.trial068.stdout: Preparing : 1/1 2026-02-22T21:34:01.425 INFO:teuthology.orchestra.run.trial068.stdout: Installing : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:34:01.487 INFO:teuthology.orchestra.run.trial068.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout: Verifying : rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout:Installed: 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout: rbd-nbd-debuginfo-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:01.546 INFO:teuthology.orchestra.run.trial068.stdout:Complete! 2026-02-22T21:34:01.573 DEBUG:teuthology.parallel:result is None 2026-02-22T21:34:01.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:34:01.715 DEBUG:teuthology.orchestra.run.trial051:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-22T21:34:01.735 INFO:teuthology.orchestra.run.trial051.stdout:19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:01.736 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:01.736 INFO:teuthology.task.install:The correct ceph version 19.2.3-689.gf90f4eae is installed. 2026-02-22T21:34:01.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:34:01.883 DEBUG:teuthology.orchestra.run.trial068:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-22T21:34:01.903 INFO:teuthology.orchestra.run.trial068.stdout:19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:01.903 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:01.903 INFO:teuthology.task.install:The correct ceph version 19.2.3-689.gf90f4eae is installed. 2026-02-22T21:34:01.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:34:02.038 DEBUG:teuthology.orchestra.run.trial125:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-22T21:34:02.059 INFO:teuthology.orchestra.run.trial125.stdout:19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:02.059 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:02.059 INFO:teuthology.task.install:The correct ceph version 19.2.3-689.gf90f4eae is installed. 2026-02-22T21:34:02.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:34:02.204 DEBUG:teuthology.orchestra.run.trial140:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-22T21:34:02.224 INFO:teuthology.orchestra.run.trial140.stdout:19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:02.224 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689.gf90f4eae.el9 2026-02-22T21:34:02.225 INFO:teuthology.task.install:The correct ceph version 19.2.3-689.gf90f4eae is installed. 2026-02-22T21:34:02.227 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T21:34:02.227 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:02.228 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T21:34:02.253 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:02.253 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T21:34:02.279 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:02.279 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T21:34:02.304 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:02.304 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T21:34:02.329 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T21:34:02.329 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:02.329 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T21:34:02.355 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T21:34:02.418 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:02.418 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T21:34:02.443 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T21:34:02.506 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:02.506 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T21:34:02.531 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T21:34:02.596 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:02.596 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T21:34:02.621 DEBUG:teuthology.orchestra.run.trial140:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T21:34:02.685 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T21:34:02.685 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:02.685 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T21:34:02.709 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T21:34:02.773 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:02.773 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T21:34:02.798 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T21:34:02.861 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:02.861 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T21:34:02.886 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T21:34:02.949 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:02.949 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T21:34:02.992 DEBUG:teuthology.orchestra.run.trial140:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T21:34:03.064 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T21:34:03.065 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:03.065 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T21:34:03.089 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T21:34:03.154 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:03.154 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T21:34:03.178 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T21:34:03.242 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:03.242 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T21:34:03.268 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T21:34:03.330 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:03.330 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T21:34:03.355 DEBUG:teuthology.orchestra.run.trial140:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T21:34:03.419 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T21:34:03.529 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T21:34:03.529 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 777 /var/log/ceph 2026-02-22T21:34:03.531 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 777 /var/log/ceph 2026-02-22T21:34:03.533 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod 777 /var/log/ceph 2026-02-22T21:34:03.535 DEBUG:teuthology.orchestra.run.trial140:> sudo chmod 777 /var/log/ceph 2026-02-22T21:34:03.559 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T21:34:03.559 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T21:34:03.597 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T21:34:03.600 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T21:34:03.602 DEBUG:teuthology.orchestra.run.trial140:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T21:34:03.625 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T21:34:03.625 DEBUG:teuthology.orchestra.run.trial051:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T21:34:03.661 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T21:34:03.663 DEBUG:teuthology.orchestra.run.trial125:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T21:34:03.665 DEBUG:teuthology.orchestra.run.trial140:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T21:34:03.703 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-22T21:34:03.703 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 240, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', 'OSD bench result'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T21:34:03.703 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rados/verify/{centos_latest ceph clusters/fixed-4 d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '64029', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 240, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', 'OSD bench result'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'stop-daemons-timeout': 1200, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5907, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial051.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZkZJqwfqN94NjV/kp3TXRAPlqgadxZzATQonuCpSVvpKHwer3AYOzAYa2UDQuY/HywWJlQjINSLKaenAe4MXs=', 'trial068.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKr+hv4I49yU6HT5IHdc3OZKdUXBoqwRpYG7ScRKM+2nhaaZgo0worr5sH10dw141BaER3bmx5aSUpqx1euzJtA=', 'trial125.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMG6J4x4AMod3GhuIGYLym1ttPAQL3O6gC+eNb4hFz6B0PjZ7XjHcLGK2phheE00Xp9JhqNtwVoUXzwPym9/3/0=', 'trial140.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHixoq3LGQW2TB0fUQ+hOlmy4Wu0sLaq6pcu+ALrX0+2zRspvWPcV5BpQq/s5LQFNUrxG5irsEfRUB0vJgJ96Nk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["-- Don't run this test on tiny clusters\nif py_len(yaml.roles) < 4 then reject() end\n"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T21:34:03.704 DEBUG:teuthology.orchestra.run.trial051:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T21:34:03.726 DEBUG:teuthology.orchestra.run.trial068:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T21:34:03.728 DEBUG:teuthology.orchestra.run.trial125:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T21:34:03.731 DEBUG:teuthology.orchestra.run.trial140:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T21:34:03.747 DEBUG:teuthology.orchestra.run.trial051:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T21:34:03.781 DEBUG:teuthology.orchestra.run.trial068:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T21:34:03.784 DEBUG:teuthology.orchestra.run.trial125:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T21:34:03.787 DEBUG:teuthology.orchestra.run.trial140:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T21:34:03.812 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:03.812 DEBUG:teuthology.orchestra.run.trial051:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T21:34:03.862 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T21:34:03.863 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_1 2026-02-22T21:34:03.918 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1308 Links: 1 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-22 21:28:19.364227561 +0000 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:03.919 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:03.919 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T21:34:03.982 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-22T21:34:03.982 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-22T21:34:03.982 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000173973 s, 2.9 MB/s 2026-02-22T21:34:03.983 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T21:34:04.042 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_2 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1292 Links: 1 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-22 21:28:19.364227561 +0000 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-22 21:27:38.765125531 +0000 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-22 21:27:38.765125531 +0000 2026-02-22T21:34:04.097 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-22 21:27:38.765125531 +0000 2026-02-22T21:34:04.097 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T21:34:04.159 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-22T21:34:04.159 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-22T21:34:04.159 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000135512 s, 3.8 MB/s 2026-02-22T21:34:04.160 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T21:34:04.217 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_3 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1296 Links: 1 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-22 21:28:19.365227540 +0000 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.275 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.275 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T21:34:04.338 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-22T21:34:04.338 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-22T21:34:04.338 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000142895 s, 3.6 MB/s 2026-02-22T21:34:04.339 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T21:34:04.396 DEBUG:teuthology.orchestra.run.trial051:> stat /dev/vg_nvme/lv_4 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Access: 2026-02-22 21:28:19.365227540 +0000 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Modify: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout:Change: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.452 INFO:teuthology.orchestra.run.trial051.stdout: Birth: 2026-02-22 21:27:38.766125509 +0000 2026-02-22T21:34:04.452 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T21:34:04.514 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records in 2026-02-22T21:34:04.514 INFO:teuthology.orchestra.run.trial051.stderr:1+0 records out 2026-02-22T21:34:04.514 INFO:teuthology.orchestra.run.trial051.stderr:512 bytes copied, 0.000143296 s, 3.6 MB/s 2026-02-22T21:34:04.515 DEBUG:teuthology.orchestra.run.trial051:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T21:34:04.571 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:04.571 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:04.571 DEBUG:teuthology.orchestra.run.trial068:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T21:34:04.588 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T21:34:04.588 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_1 2026-02-22T21:34:04.644 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T21:34:04.644 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-22 21:28:19.535345367 +0000 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.645 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.646 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T21:34:04.708 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-22T21:34:04.708 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-22T21:34:04.708 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000149952 s, 3.4 MB/s 2026-02-22T21:34:04.709 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T21:34:04.765 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_2 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-22 21:28:19.536345351 +0000 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.821 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:04.821 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T21:34:04.884 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-22T21:34:04.884 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-22T21:34:04.884 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000192041 s, 2.7 MB/s 2026-02-22T21:34:04.885 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T21:34:04.942 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_3 2026-02-22T21:34:04.998 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T21:34:04.998 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1318 Links: 1 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-22 21:28:19.536345351 +0000 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-22 21:27:38.855998653 +0000 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-22 21:27:38.855998653 +0000 2026-02-22T21:34:04.999 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-22 21:27:38.855998653 +0000 2026-02-22T21:34:05.000 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T21:34:05.062 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-22T21:34:05.062 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-22T21:34:05.062 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000150643 s, 3.4 MB/s 2026-02-22T21:34:05.063 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T21:34:05.121 DEBUG:teuthology.orchestra.run.trial068:> stat /dev/vg_nvme/lv_4 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Access: 2026-02-22 21:28:19.536345351 +0000 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Modify: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout:Change: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:05.177 INFO:teuthology.orchestra.run.trial068.stdout: Birth: 2026-02-22 21:27:38.854998670 +0000 2026-02-22T21:34:05.177 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T21:34:05.240 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records in 2026-02-22T21:34:05.240 INFO:teuthology.orchestra.run.trial068.stderr:1+0 records out 2026-02-22T21:34:05.240 INFO:teuthology.orchestra.run.trial068.stderr:512 bytes copied, 0.000129844 s, 3.9 MB/s 2026-02-22T21:34:05.241 DEBUG:teuthology.orchestra.run.trial068:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T21:34:05.297 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:05.297 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:05.297 DEBUG:teuthology.orchestra.run.trial125:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T21:34:05.315 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T21:34:05.315 DEBUG:teuthology.orchestra.run.trial125:> stat /dev/vg_nvme/lv_1 2026-02-22T21:34:05.371 INFO:teuthology.orchestra.run.trial125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T21:34:05.371 INFO:teuthology.orchestra.run.trial125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:05.371 INFO:teuthology.orchestra.run.trial125.stdout:Device: 6h/6d Inode: 1300 Links: 1 2026-02-22T21:34:05.371 INFO:teuthology.orchestra.run.trial125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:05.371 INFO:teuthology.orchestra.run.trial125.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:05.372 INFO:teuthology.orchestra.run.trial125.stdout:Access: 2026-02-22 21:28:15.228623333 +0000 2026-02-22T21:34:05.372 INFO:teuthology.orchestra.run.trial125.stdout:Modify: 2026-02-22 21:27:39.985249550 +0000 2026-02-22T21:34:05.372 INFO:teuthology.orchestra.run.trial125.stdout:Change: 2026-02-22 21:27:39.985249550 +0000 2026-02-22T21:34:05.372 INFO:teuthology.orchestra.run.trial125.stdout: Birth: 2026-02-22 21:27:39.985249550 +0000 2026-02-22T21:34:05.372 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T21:34:05.435 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records in 2026-02-22T21:34:05.435 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records out 2026-02-22T21:34:05.436 INFO:teuthology.orchestra.run.trial125.stderr:512 bytes copied, 0.000163203 s, 3.1 MB/s 2026-02-22T21:34:05.437 DEBUG:teuthology.orchestra.run.trial125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T21:34:05.496 DEBUG:teuthology.orchestra.run.trial125:> stat /dev/vg_nvme/lv_2 2026-02-22T21:34:05.552 INFO:teuthology.orchestra.run.trial125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Access: 2026-02-22 21:28:15.228623333 +0000 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Modify: 2026-02-22 21:27:39.984249568 +0000 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout:Change: 2026-02-22 21:27:39.985249550 +0000 2026-02-22T21:34:05.553 INFO:teuthology.orchestra.run.trial125.stdout: Birth: 2026-02-22 21:27:39.984249568 +0000 2026-02-22T21:34:05.554 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T21:34:05.617 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records in 2026-02-22T21:34:05.618 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records out 2026-02-22T21:34:05.618 INFO:teuthology.orchestra.run.trial125.stderr:512 bytes copied, 0.000141252 s, 3.6 MB/s 2026-02-22T21:34:05.619 DEBUG:teuthology.orchestra.run.trial125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T21:34:05.675 DEBUG:teuthology.orchestra.run.trial125:> stat /dev/vg_nvme/lv_3 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Device: 6h/6d Inode: 1313 Links: 1 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Access: 2026-02-22 21:28:15.228623333 +0000 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Modify: 2026-02-22 21:27:39.986249532 +0000 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout:Change: 2026-02-22 21:27:39.986249532 +0000 2026-02-22T21:34:05.732 INFO:teuthology.orchestra.run.trial125.stdout: Birth: 2026-02-22 21:27:39.986249532 +0000 2026-02-22T21:34:05.732 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T21:34:05.794 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records in 2026-02-22T21:34:05.794 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records out 2026-02-22T21:34:05.794 INFO:teuthology.orchestra.run.trial125.stderr:512 bytes copied, 0.000168923 s, 3.0 MB/s 2026-02-22T21:34:05.795 DEBUG:teuthology.orchestra.run.trial125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T21:34:05.852 DEBUG:teuthology.orchestra.run.trial125:> stat /dev/vg_nvme/lv_4 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Device: 6h/6d Inode: 1288 Links: 1 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Access: 2026-02-22 21:28:15.229623315 +0000 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Modify: 2026-02-22 21:27:39.984249568 +0000 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout:Change: 2026-02-22 21:27:39.984249568 +0000 2026-02-22T21:34:05.908 INFO:teuthology.orchestra.run.trial125.stdout: Birth: 2026-02-22 21:27:39.984249568 +0000 2026-02-22T21:34:05.908 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T21:34:05.973 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records in 2026-02-22T21:34:05.973 INFO:teuthology.orchestra.run.trial125.stderr:1+0 records out 2026-02-22T21:34:05.973 INFO:teuthology.orchestra.run.trial125.stderr:512 bytes copied, 0.000148235 s, 3.5 MB/s 2026-02-22T21:34:05.974 DEBUG:teuthology.orchestra.run.trial125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T21:34:06.032 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:06.032 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:06.032 DEBUG:teuthology.orchestra.run.trial140:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T21:34:06.050 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T21:34:06.050 DEBUG:teuthology.orchestra.run.trial140:> stat /dev/vg_nvme/lv_1 2026-02-22T21:34:06.109 INFO:teuthology.orchestra.run.trial140.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Access: 2026-02-22 21:28:16.409758724 +0000 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Modify: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout:Change: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.110 INFO:teuthology.orchestra.run.trial140.stdout: Birth: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.111 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T21:34:06.172 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records in 2026-02-22T21:34:06.172 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records out 2026-02-22T21:34:06.172 INFO:teuthology.orchestra.run.trial140.stderr:512 bytes copied, 0.00016148 s, 3.2 MB/s 2026-02-22T21:34:06.173 DEBUG:teuthology.orchestra.run.trial140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T21:34:06.230 DEBUG:teuthology.orchestra.run.trial140:> stat /dev/vg_nvme/lv_2 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout:Access: 2026-02-22 21:28:16.410758713 +0000 2026-02-22T21:34:06.285 INFO:teuthology.orchestra.run.trial140.stdout:Modify: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.286 INFO:teuthology.orchestra.run.trial140.stdout:Change: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.286 INFO:teuthology.orchestra.run.trial140.stdout: Birth: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.286 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T21:34:06.350 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records in 2026-02-22T21:34:06.350 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records out 2026-02-22T21:34:06.350 INFO:teuthology.orchestra.run.trial140.stderr:512 bytes copied, 0.000130853 s, 3.9 MB/s 2026-02-22T21:34:06.351 DEBUG:teuthology.orchestra.run.trial140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T21:34:06.407 DEBUG:teuthology.orchestra.run.trial140:> stat /dev/vg_nvme/lv_3 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout:Access: 2026-02-22 21:28:16.410758713 +0000 2026-02-22T21:34:06.465 INFO:teuthology.orchestra.run.trial140.stdout:Modify: 2026-02-22 21:27:38.553239016 +0000 2026-02-22T21:34:06.466 INFO:teuthology.orchestra.run.trial140.stdout:Change: 2026-02-22 21:27:38.553239016 +0000 2026-02-22T21:34:06.466 INFO:teuthology.orchestra.run.trial140.stdout: Birth: 2026-02-22 21:27:38.553239016 +0000 2026-02-22T21:34:06.466 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T21:34:06.529 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records in 2026-02-22T21:34:06.529 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records out 2026-02-22T21:34:06.529 INFO:teuthology.orchestra.run.trial140.stderr:512 bytes copied, 0.000146412 s, 3.5 MB/s 2026-02-22T21:34:06.530 DEBUG:teuthology.orchestra.run.trial140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T21:34:06.587 DEBUG:teuthology.orchestra.run.trial140:> stat /dev/vg_nvme/lv_4 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Context: system_u:object_r:device_t:s0 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Access: 2026-02-22 21:28:16.410758713 +0000 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Modify: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.644 INFO:teuthology.orchestra.run.trial140.stdout:Change: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.645 INFO:teuthology.orchestra.run.trial140.stdout: Birth: 2026-02-22 21:27:38.554238990 +0000 2026-02-22T21:34:06.645 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T21:34:06.709 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records in 2026-02-22T21:34:06.709 INFO:teuthology.orchestra.run.trial140.stderr:1+0 records out 2026-02-22T21:34:06.710 INFO:teuthology.orchestra.run.trial140.stderr:512 bytes copied, 0.000144699 s, 3.5 MB/s 2026-02-22T21:34:06.711 DEBUG:teuthology.orchestra.run.trial140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T21:34:06.769 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:06.770 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial051.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial125.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial140.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2026-02-22T21:34:06.770 INFO:tasks.ceph:Generating config... 2026-02-22T21:34:06.772 INFO:tasks.ceph:[client] debug monc = 20 2026-02-22T21:34:06.773 INFO:tasks.ceph:[client] debug ms = 1 2026-02-22T21:34:06.773 INFO:tasks.ceph:[client] debug objecter = 20 2026-02-22T21:34:06.773 INFO:tasks.ceph:[client] debug rados = 20 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] ms type = async 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] osd heartbeat grace = 160 2026-02-22T21:34:06.773 INFO:tasks.ceph:[global] osd op thread timeout = 150 2026-02-22T21:34:06.773 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2026-02-22T21:34:06.774 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug bluefs = 1 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug bluestore = 1 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T21:34:06.775 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd class default list = * 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd class load list = * 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd client watch timeout = 240 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd fast shutdown = False 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-22T21:34:06.776 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-22T21:34:06.777 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-22T21:34:06.777 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T21:34:06.777 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-22T21:34:06.777 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-22T21:34:06.777 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-22T21:34:06.777 INFO:tasks.ceph:Setting up mon.a... 2026-02-22T21:34:06.777 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-22T21:34:06.813 INFO:teuthology.orchestra.run.trial051.stdout:creating /etc/ceph/ceph.keyring 2026-02-22T21:34:06.816 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-22T21:34:06.892 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T21:34:06.959 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.51'), ('mon.b', '10.20.193.68'), ('mon.c', '10.20.193.125')] 2026-02-22T21:34:06.960 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.51,10.20.193.68,10.20.193.125', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 160, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 240, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-22T21:34:06.962 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:06.962 DEBUG:teuthology.orchestra.run.trial051:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T21:34:07.015 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.51 --add b 10.20.193.68 --add c 10.20.193.125 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:monmaptool: generated fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:setting min_mon_release = quincy 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:epoch 0 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:min_mon_release 17 (quincy) 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:election_strategy: 1 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:34:07.090 INFO:teuthology.orchestra.run.trial051.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-22T21:34:07.091 DEBUG:teuthology.orchestra.run.trial051:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T21:34:07.148 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f64be66c-6ce4-4d13-a1b8-cf6040bfed5c... 2026-02-22T21:34:07.149 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T21:34:07.190 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T21:34:07.192 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T21:34:07.194 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout:[global] 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: chdir = "" 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: auth supported = cephx 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: filestore xattr use omap = true 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: mon clock drift allowed = 1.000 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: osd crush chooseleaf type = 0 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: auth debug = true 2026-02-22T21:34:07.227 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: ms die on old message = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: ms die on bug = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon pg warn max object skew = 0 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd pool default size = 2 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon osd allow primary affinity = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon osd allow pg remap = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on legacy crush tunables = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on crush straw calc version zero = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on no sortbitwise = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on osd down out interval zero = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on too few osds = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon_allow_pool_size_one = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd default data pool replay window = 5 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon allow pool delete = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon cluster log file level = debug 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: debug asserts on shutdown = true 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon health detail to clog = false 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon host = "10.20.193.51,10.20.193.68,10.20.193.125" 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon client directed command retry = 5 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: mon election default strategy = 1 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: ms inject socket failures = 5000 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: ms type = async 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd heartbeat grace = 160 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd op thread timeout = 150 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: fsid = f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout:[osd] 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd journal size = 100 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.228 INFO:teuthology.orchestra.run.trial051.stdout: osd scrub load threshold = 5.0 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd scrub max interval = 600 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd mclock profile = high_recovery_ops 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd mclock skip benchmark = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd recover clone overlap = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd recovery max chunk = 1048576 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug shutdown = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug op order = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug verify stray on activate = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug trim objects = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd open classes on start = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug pg log writeout = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd deep scrub update digest min age = 30 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd map max advance = 10 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: journal zero on create = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: filestore ondisk finisher threads = 3 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: filestore apply finisher threads = 3 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bdev debug aio = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug misdirected ops = true 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bdev async discard = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bdev enable discard = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bluestore allocator = bitmap 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bluestore block size = 96636764160 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bluestore fsck on mount = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: bluestore zero block detection = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug bluefs = 1 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug bluestore = 1 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug monc = 20 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug ms = 1 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug osd = 20 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: debug rocksdb = 10 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: mon osd full ratio = 0.9 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: mon osd nearfull ratio = 0.8 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd class default list = * 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd class load list = * 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd client watch timeout = 240 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug verify cached snaps = True 2026-02-22T21:34:07.229 INFO:teuthology.orchestra.run.trial051.stdout: osd debug verify missing on start = True 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd failsafe full ratio = 0.95 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd fast shutdown = False 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd mclock override recovery settings = True 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd objectstore = bluestore 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd op queue = debug_random 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd op queue cut off = debug_random 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: osd shutdown pgref assert = True 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout:[mgr] 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug ms = 1 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug mgr = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug mon = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug auth = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mgr/telemetry/nag = false 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout:[mon] 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug ms = 1 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug mon = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug paxos = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: debug auth = 20 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon data avail warn = 5 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon mgr mkfs grace = 240 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon osd reporter subtree level = osd 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon osd prime pg temp = true 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: # 1m isn't quite enough 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon_down_mkfs_grace = 2m 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.230 INFO:teuthology.orchestra.run.trial051.stdout: mon_warn_on_filestore_osds = false 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon min osdmap epochs = 50 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon osd crush smoke test = False 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon osdmap full prune interval = 2 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon osdmap full prune min = 15 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon osdmap full prune txsize = 2 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon scrub interval = 300 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: mon warn on pool no app = False 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: paxos service trim min = 10 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout:[client] 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: rgw cache enabled = true 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: rgw enable ops log = true 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: rgw enable usage log = true 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: debug monc = 20 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: debug ms = 1 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: debug objecter = 20 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout: debug rados = 20 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout:[mon.a] 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout:[mon.b] 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial051.stdout:[mon.c] 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout:[global] 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: chdir = "" 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: auth supported = cephx 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: filestore xattr use omap = true 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: mon clock drift allowed = 1.000 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.231 INFO:teuthology.orchestra.run.trial068.stdout: osd crush chooseleaf type = 0 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: auth debug = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: ms die on old message = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: ms die on bug = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon pg warn max object skew = 0 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default size = 2 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow primary affinity = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon osd allow pg remap = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on legacy crush tunables = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on crush straw calc version zero = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on no sortbitwise = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on osd down out interval zero = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on too few osds = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon_allow_pool_size_one = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd default data pool replay window = 5 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon allow pool delete = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon cluster log file level = debug 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: debug asserts on shutdown = true 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon health detail to clog = false 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon host = "10.20.193.51,10.20.193.68,10.20.193.125" 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon client directed command retry = 5 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: mon election default strategy = 1 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: ms inject socket failures = 5000 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: ms type = async 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd heartbeat grace = 160 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: osd op thread timeout = 150 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: fsid = f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.232 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout:[osd] 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd journal size = 100 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub load threshold = 5.0 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd scrub max interval = 600 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock profile = high_recovery_ops 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock skip benchmark = True 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd recover clone overlap = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd recovery max chunk = 1048576 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug shutdown = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug op order = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify stray on activate = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug trim objects = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd open classes on start = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug pg log writeout = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd deep scrub update digest min age = 30 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd map max advance = 10 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: journal zero on create = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: filestore ondisk finisher threads = 3 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: filestore apply finisher threads = 3 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bdev debug aio = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd debug misdirected ops = true 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bdev async discard = True 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bdev enable discard = True 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bluestore allocator = bitmap 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bluestore block size = 96636764160 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bluestore fsck on mount = True 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: bluestore zero block detection = True 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug bluefs = 1 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug bluestore = 1 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug monc = 20 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug osd = 20 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: debug rocksdb = 10 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: mon osd full ratio = 0.9 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: mon osd nearfull ratio = 0.8 2026-02-22T21:34:07.233 INFO:teuthology.orchestra.run.trial068.stdout: osd class default list = * 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd class load list = * 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd client watch timeout = 240 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify cached snaps = True 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd debug verify missing on start = True 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd failsafe full ratio = 0.95 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd fast shutdown = False 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd mclock override recovery settings = True 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd objectstore = bluestore 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd op queue = debug_random 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd op queue cut off = debug_random 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: osd shutdown pgref assert = True 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout:[mgr] 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug mgr = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mgr/telemetry/nag = false 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout:[mon] 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug mon = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug paxos = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: debug auth = 20 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon data avail warn = 5 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon mgr mkfs grace = 240 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon osd reporter subtree level = osd 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon osd prime pg temp = true 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: # 1m isn't quite enough 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon_down_mkfs_grace = 2m 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon_warn_on_filestore_osds = false 2026-02-22T21:34:07.234 INFO:teuthology.orchestra.run.trial068.stdout: mon min osdmap epochs = 50 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon osd crush smoke test = False 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune interval = 2 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune min = 15 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon osdmap full prune txsize = 2 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon scrub interval = 300 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: mon warn on pool no app = False 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: paxos service trim min = 10 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout:[client] 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: rgw cache enabled = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable ops log = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: rgw enable usage log = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: debug monc = 20 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: debug ms = 1 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: debug objecter = 20 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout: debug rados = 20 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout:[mon.a] 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout:[mon.b] 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial068.stdout:[mon.c] 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout:[global] 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: chdir = "" 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: auth supported = cephx 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: filestore xattr use omap = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: mon clock drift allowed = 1.000 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: osd crush chooseleaf type = 0 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: auth debug = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: ms die on old message = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: ms die on bug = true 2026-02-22T21:34:07.235 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon pg warn max object skew = 0 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd pool default size = 2 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon osd allow primary affinity = true 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon osd allow pg remap = true 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on legacy crush tunables = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on crush straw calc version zero = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on no sortbitwise = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on osd down out interval zero = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on too few osds = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon_allow_pool_size_one = true 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd default data pool replay window = 5 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon allow pool delete = true 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon cluster log file level = debug 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: debug asserts on shutdown = true 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon health detail to clog = false 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon host = "10.20.193.51,10.20.193.68,10.20.193.125" 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon client directed command retry = 5 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: mon election default strategy = 1 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: ms inject socket failures = 5000 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: ms type = async 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd heartbeat grace = 160 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd op thread timeout = 150 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: fsid = f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout:[osd] 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd journal size = 100 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd scrub load threshold = 5.0 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd scrub max interval = 600 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd mclock profile = high_recovery_ops 2026-02-22T21:34:07.236 INFO:teuthology.orchestra.run.trial125.stdout: osd mclock skip benchmark = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd recover clone overlap = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd recovery max chunk = 1048576 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug shutdown = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug op order = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug verify stray on activate = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug trim objects = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd open classes on start = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug pg log writeout = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd deep scrub update digest min age = 30 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd map max advance = 10 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: journal zero on create = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: filestore ondisk finisher threads = 3 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: filestore apply finisher threads = 3 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bdev debug aio = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug misdirected ops = true 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bdev async discard = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bdev enable discard = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bluestore allocator = bitmap 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bluestore block size = 96636764160 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bluestore fsck on mount = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: bluestore zero block detection = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug bluefs = 1 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug bluestore = 1 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug monc = 20 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug ms = 1 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug osd = 20 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: debug rocksdb = 10 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: mon osd full ratio = 0.9 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: mon osd nearfull ratio = 0.8 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd class default list = * 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd class load list = * 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd client watch timeout = 240 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug verify cached snaps = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd debug verify missing on start = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd failsafe full ratio = 0.95 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd fast shutdown = False 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd mclock override recovery settings = True 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd objectstore = bluestore 2026-02-22T21:34:07.237 INFO:teuthology.orchestra.run.trial125.stdout: osd op queue = debug_random 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: osd op queue cut off = debug_random 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: osd shutdown pgref assert = True 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout:[mgr] 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug ms = 1 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug mgr = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug mon = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug auth = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mgr/telemetry/nag = false 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout:[mon] 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug ms = 1 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug mon = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug paxos = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: debug auth = 20 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon data avail warn = 5 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon mgr mkfs grace = 240 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osd reporter subtree level = osd 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osd prime pg temp = true 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: # 1m isn't quite enough 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon_down_mkfs_grace = 2m 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon_warn_on_filestore_osds = false 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon min osdmap epochs = 50 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osd crush smoke test = False 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osdmap full prune interval = 2 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osdmap full prune min = 15 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon osdmap full prune txsize = 2 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon scrub interval = 300 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: mon warn on pool no app = False 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: paxos service trim min = 10 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout: 2026-02-22T21:34:07.238 INFO:teuthology.orchestra.run.trial125.stdout:[client] 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: rgw cache enabled = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: rgw enable ops log = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: rgw enable usage log = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: debug monc = 20 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: debug ms = 1 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: debug objecter = 20 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout: debug rados = 20 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout:[mon.a] 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout:[mon.b] 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial125.stdout:[mon.c] 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout:[global] 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: chdir = "" 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: auth supported = cephx 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: filestore xattr use omap = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: mon clock drift allowed = 1.000 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: osd crush chooseleaf type = 0 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: auth debug = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: ms die on old message = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: ms die on bug = true 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: mon pg warn max object skew = 0 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.239 INFO:teuthology.orchestra.run.trial140.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd pool default size = 2 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon osd allow primary affinity = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon osd allow pg remap = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on legacy crush tunables = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on crush straw calc version zero = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on no sortbitwise = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on osd down out interval zero = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on too few osds = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon_allow_pool_size_one = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd default data pool replay window = 5 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon allow pool delete = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon cluster log file level = debug 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: debug asserts on shutdown = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon health detail to clog = false 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon host = "10.20.193.51,10.20.193.68,10.20.193.125" 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon client directed command retry = 5 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: mon election default strategy = 1 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: ms inject socket failures = 5000 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: ms type = async 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd heartbeat grace = 160 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd op thread timeout = 150 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: fsid = f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout:[osd] 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd journal size = 100 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd scrub load threshold = 5.0 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd scrub max interval = 600 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd mclock profile = high_recovery_ops 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd mclock skip benchmark = True 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd recover clone overlap = true 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd recovery max chunk = 1048576 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.240 INFO:teuthology.orchestra.run.trial140.stdout: osd debug shutdown = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug op order = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug verify stray on activate = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug trim objects = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd open classes on start = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug pg log writeout = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd deep scrub update digest min age = 30 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd map max advance = 10 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: journal zero on create = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: filestore ondisk finisher threads = 3 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: filestore apply finisher threads = 3 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bdev debug aio = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug misdirected ops = true 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bdev async discard = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bdev enable discard = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bluestore allocator = bitmap 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bluestore block size = 96636764160 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bluestore fsck on mount = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: bluestore zero block detection = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug bluefs = 1 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug bluestore = 1 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug monc = 20 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug ms = 1 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug osd = 20 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: debug rocksdb = 10 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: mon osd full ratio = 0.9 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: mon osd nearfull ratio = 0.8 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd class default list = * 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd class load list = * 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd client watch timeout = 240 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug verify cached snaps = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd debug verify missing on start = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd failsafe full ratio = 0.95 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd fast shutdown = False 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd mclock override recovery settings = True 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd objectstore = bluestore 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd op queue = debug_random 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd op queue cut off = debug_random 2026-02-22T21:34:07.241 INFO:teuthology.orchestra.run.trial140.stdout: osd shutdown pgref assert = True 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout:[mgr] 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug ms = 1 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug mgr = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug mon = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug auth = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mgr/telemetry/nag = false 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout:[mon] 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug ms = 1 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug mon = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug paxos = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: debug auth = 20 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon data avail warn = 5 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon mgr mkfs grace = 240 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon reweight min pgs per osd = 4 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osd reporter subtree level = osd 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osd prime pg temp = true 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon reweight min bytes per osd = 10 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: # 1m isn't quite enough 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon_down_mkfs_grace = 2m 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon_warn_on_filestore_osds = false 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon min osdmap epochs = 50 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osd crush smoke test = False 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osdmap full prune interval = 2 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osdmap full prune min = 15 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon osdmap full prune txsize = 2 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon scrub interval = 300 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: mon warn on pool no app = False 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: paxos service trim min = 10 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout:[client] 2026-02-22T21:34:07.242 INFO:teuthology.orchestra.run.trial140.stdout: rgw cache enabled = true 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: rgw enable ops log = true 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: rgw enable usage log = true 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: debug monc = 20 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: debug ms = 1 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: debug objecter = 20 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout: debug rados = 20 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout:[mon.a] 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout:[mon.b] 2026-02-22T21:34:07.243 INFO:teuthology.orchestra.run.trial140.stdout:[mon.c] 2026-02-22T21:34:07.246 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-22T21:34:07.246 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-22T21:34:07.326 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T21:34:07.326 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:07.326 DEBUG:teuthology.orchestra.run.trial051:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-22T21:34:07.381 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:07.381 DEBUG:teuthology.orchestra.run.trial051:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-22T21:34:07.437 INFO:tasks.ceph:Sending monmap to node ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:34:07.437 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:07.437 DEBUG:teuthology.orchestra.run.trial051:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T21:34:07.437 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T21:34:07.510 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:34:07.510 DEBUG:teuthology.orchestra.run.trial051:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.567 INFO:tasks.ceph:Sending monmap to node ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:34:07.567 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:07.567 DEBUG:teuthology.orchestra.run.trial068:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T21:34:07.567 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T21:34:07.602 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:34:07.602 DEBUG:teuthology.orchestra.run.trial068:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.658 INFO:tasks.ceph:Sending monmap to node ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:34:07.658 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:07.658 DEBUG:teuthology.orchestra.run.trial125:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T21:34:07.658 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T21:34:07.691 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:34:07.691 DEBUG:teuthology.orchestra.run.trial125:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.747 INFO:tasks.ceph:Sending monmap to node ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:34:07.747 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:07.747 DEBUG:teuthology.orchestra.run.trial140:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T21:34:07.748 DEBUG:teuthology.orchestra.run.trial140:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T21:34:07.781 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:34:07.781 DEBUG:teuthology.orchestra.run.trial140:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:07.838 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T21:34:07.839 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T21:34:07.839 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2026-02-22T21:34:07.884 INFO:teuthology.orchestra.run.trial051.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2026-02-22T21:34:07.887 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T21:34:07.931 INFO:teuthology.orchestra.run.trial140.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T21:34:07.933 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T21:34:07.934 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T21:34:07.934 DEBUG:teuthology.orchestra.run.trial140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-22T21:34:08.020 INFO:teuthology.orchestra.run.trial140.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-22T21:34:08.031 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T21:34:08.031 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial051.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial068.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial125.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@trial140.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2026-02-22T21:34:08.032 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-22T21:34:08.056 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:08.056 INFO:tasks.ceph:role: osd.0 2026-02-22T21:34:08.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:34:08.057 DEBUG:teuthology.orchestra.run.trial051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:08.121 INFO:teuthology.orchestra.run.trial051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:08.692 INFO:teuthology.orchestra.run.trial051.stdout:Discarding blocks...Done. 2026-02-22T21:34:08.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial051.front.sepia.ceph.com -o noatime 2026-02-22T21:34:08.694 DEBUG:teuthology.orchestra.run.trial051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-22T21:34:08.763 DEBUG:teuthology.orchestra.run.trial051:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-22T21:34:08.831 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2026-02-22T21:34:08.896 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:08.896 INFO:tasks.ceph:role: osd.4 2026-02-22T21:34:08.896 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:34:08.896 DEBUG:teuthology.orchestra.run.trial051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T21:34:08.962 INFO:teuthology.orchestra.run.trial051.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:08.963 INFO:teuthology.orchestra.run.trial051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:09.524 INFO:teuthology.orchestra.run.trial051.stdout:Discarding blocks...Done. 2026-02-22T21:34:09.525 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial051.front.sepia.ceph.com -o noatime 2026-02-22T21:34:09.526 DEBUG:teuthology.orchestra.run.trial051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2026-02-22T21:34:09.595 DEBUG:teuthology.orchestra.run.trial051:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2026-02-22T21:34:09.661 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2026-02-22T21:34:09.726 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:09.726 INFO:tasks.ceph:role: osd.8 2026-02-22T21:34:09.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:34:09.726 DEBUG:teuthology.orchestra.run.trial051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:09.790 INFO:teuthology.orchestra.run.trial051.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:09.791 INFO:teuthology.orchestra.run.trial051.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:09.791 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:09.791 INFO:teuthology.orchestra.run.trial051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:10.350 INFO:teuthology.orchestra.run.trial051.stdout:Discarding blocks...Done. 2026-02-22T21:34:10.352 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial051.front.sepia.ceph.com -o noatime 2026-02-22T21:34:10.352 DEBUG:teuthology.orchestra.run.trial051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2026-02-22T21:34:10.423 DEBUG:teuthology.orchestra.run.trial051:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2026-02-22T21:34:10.491 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2026-02-22T21:34:10.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:10.555 INFO:tasks.ceph:role: osd.12 2026-02-22T21:34:10.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:34:10.555 DEBUG:teuthology.orchestra.run.trial051:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-22T21:34:10.621 INFO:teuthology.orchestra.run.trial051.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:10.621 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:10.622 INFO:teuthology.orchestra.run.trial051.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:11.179 INFO:teuthology.orchestra.run.trial051.stdout:Discarding blocks...Done. 2026-02-22T21:34:11.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial051.front.sepia.ceph.com -o noatime 2026-02-22T21:34:11.182 DEBUG:teuthology.orchestra.run.trial051:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2026-02-22T21:34:11.251 DEBUG:teuthology.orchestra.run.trial051:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2026-02-22T21:34:11.320 DEBUG:teuthology.orchestra.run.trial051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:11.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:11.422+0000 7fc266813780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-22T21:34:11.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:11.422+0000 7fc266813780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-22T21:34:11.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:11.422+0000 7fc266813780 -1 bdev(0x562d1064f800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:11.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:11.422+0000 7fc266813780 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-22T21:34:15.333 DEBUG:teuthology.orchestra.run.trial051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-22T21:34:15.397 DEBUG:teuthology.orchestra.run.trial051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:15.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:15.476+0000 7f16aac23780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2026-02-22T21:34:15.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:15.476+0000 7f16aac23780 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2026-02-22T21:34:15.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:15.476+0000 7f16aac23780 -1 bdev(0x55a585741800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:15.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:15.476+0000 7f16aac23780 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2026-02-22T21:34:19.364 DEBUG:teuthology.orchestra.run.trial051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2026-02-22T21:34:19.429 DEBUG:teuthology.orchestra.run.trial051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:19.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:19.504+0000 7f3acae0f780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2026-02-22T21:34:19.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:19.504+0000 7f3acae0f780 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2026-02-22T21:34:19.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:19.505+0000 7f3acae0f780 -1 bdev(0x5563bc9dd800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:19.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:19.505+0000 7f3acae0f780 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2026-02-22T21:34:23.386 DEBUG:teuthology.orchestra.run.trial051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2026-02-22T21:34:23.451 DEBUG:teuthology.orchestra.run.trial051:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:23.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:23.526+0000 7faa08f10780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2026-02-22T21:34:23.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:23.527+0000 7faa08f10780 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2026-02-22T21:34:23.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:23.527+0000 7faa08f10780 -1 bdev(0x55d625e55800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:23.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:34:23.527+0000 7faa08f10780 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2026-02-22T21:34:27.406 DEBUG:teuthology.orchestra.run.trial051:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2026-02-22T21:34:27.472 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-22T21:34:27.499 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:27.499 INFO:tasks.ceph:role: osd.1 2026-02-22T21:34:27.499 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:34:27.500 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T21:34:27.563 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:27.563 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:27.564 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:28.133 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-22T21:34:28.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-22T21:34:28.136 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2026-02-22T21:34:28.202 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-22T21:34:28.268 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2026-02-22T21:34:28.330 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:28.331 INFO:tasks.ceph:role: osd.5 2026-02-22T21:34:28.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:34:28.331 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T21:34:28.397 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:28.397 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:28.397 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:28.398 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:28.956 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-22T21:34:28.958 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-22T21:34:28.958 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2026-02-22T21:34:29.032 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2026-02-22T21:34:29.098 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2026-02-22T21:34:29.163 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:29.163 INFO:tasks.ceph:role: osd.9 2026-02-22T21:34:29.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:34:29.163 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T21:34:29.227 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:29.227 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:29.227 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:29.227 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:29.227 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:29.228 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:29.228 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:29.228 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:29.228 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:29.228 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:29.785 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-22T21:34:29.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-22T21:34:29.787 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2026-02-22T21:34:29.862 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2026-02-22T21:34:29.930 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2026-02-22T21:34:29.994 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:29.994 INFO:tasks.ceph:role: osd.13 2026-02-22T21:34:29.994 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:34:29.994 DEBUG:teuthology.orchestra.run.trial068:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-22T21:34:30.060 INFO:teuthology.orchestra.run.trial068.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:30.060 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:30.060 INFO:teuthology.orchestra.run.trial068.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:30.060 INFO:teuthology.orchestra.run.trial068.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:30.060 INFO:teuthology.orchestra.run.trial068.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:30.061 INFO:teuthology.orchestra.run.trial068.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:30.061 INFO:teuthology.orchestra.run.trial068.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:30.061 INFO:teuthology.orchestra.run.trial068.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:30.061 INFO:teuthology.orchestra.run.trial068.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:30.061 INFO:teuthology.orchestra.run.trial068.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:30.617 INFO:teuthology.orchestra.run.trial068.stdout:Discarding blocks...Done. 2026-02-22T21:34:30.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial068.front.sepia.ceph.com -o noatime 2026-02-22T21:34:30.620 DEBUG:teuthology.orchestra.run.trial068:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2026-02-22T21:34:30.696 DEBUG:teuthology.orchestra.run.trial068:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2026-02-22T21:34:30.764 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:30.845 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:30.839+0000 7fb0685a3780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-22T21:34:30.845 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:30.839+0000 7fb0685a3780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-22T21:34:30.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:30.839+0000 7fb0685a3780 -1 bdev(0x55b555431800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:30.846 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:30.840+0000 7fb0685a3780 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-22T21:34:34.730 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-22T21:34:34.795 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:34.871 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:34.866+0000 7fd43d02c780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2026-02-22T21:34:34.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:34.867+0000 7fd43d02c780 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2026-02-22T21:34:34.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:34.867+0000 7fd43d02c780 -1 bdev(0x5582aa19d800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:34.872 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:34.867+0000 7fd43d02c780 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2026-02-22T21:34:38.768 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2026-02-22T21:34:38.833 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:38.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:38.906+0000 7ffb17011780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2026-02-22T21:34:38.911 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:38.906+0000 7ffb17011780 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2026-02-22T21:34:38.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:38.906+0000 7ffb17011780 -1 bdev(0x558ef48e3800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:38.912 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:38.906+0000 7ffb17011780 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2026-02-22T21:34:42.788 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2026-02-22T21:34:42.853 DEBUG:teuthology.orchestra.run.trial068:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:42.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:42.926+0000 7ff6f960c780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2026-02-22T21:34:42.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:42.926+0000 7ff6f960c780 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2026-02-22T21:34:42.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:42.926+0000 7ff6f960c780 -1 bdev(0x56051003f800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:42.931 INFO:teuthology.orchestra.run.trial068.stderr:2026-02-22T21:34:42.926+0000 7ff6f960c780 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2026-02-22T21:34:46.809 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2026-02-22T21:34:46.874 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-22T21:34:46.900 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:46.900 INFO:tasks.ceph:role: osd.2 2026-02-22T21:34:46.900 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:34:46.900 DEBUG:teuthology.orchestra.run.trial125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T21:34:46.965 INFO:teuthology.orchestra.run.trial125.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:46.966 INFO:teuthology.orchestra.run.trial125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:47.538 INFO:teuthology.orchestra.run.trial125.stdout:Discarding blocks...Done. 2026-02-22T21:34:47.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial125.front.sepia.ceph.com -o noatime 2026-02-22T21:34:47.540 DEBUG:teuthology.orchestra.run.trial125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-22T21:34:47.608 DEBUG:teuthology.orchestra.run.trial125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-22T21:34:47.676 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2026-02-22T21:34:47.741 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:47.741 INFO:tasks.ceph:role: osd.6 2026-02-22T21:34:47.741 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:34:47.741 DEBUG:teuthology.orchestra.run.trial125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:47.807 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:47.808 INFO:teuthology.orchestra.run.trial125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:48.378 INFO:teuthology.orchestra.run.trial125.stdout:Discarding blocks...Done. 2026-02-22T21:34:48.380 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial125.front.sepia.ceph.com -o noatime 2026-02-22T21:34:48.380 DEBUG:teuthology.orchestra.run.trial125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2026-02-22T21:34:48.448 DEBUG:teuthology.orchestra.run.trial125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2026-02-22T21:34:48.514 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2026-02-22T21:34:48.580 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:48.580 INFO:tasks.ceph:role: osd.10 2026-02-22T21:34:48.580 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:34:48.580 DEBUG:teuthology.orchestra.run.trial125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T21:34:48.645 INFO:teuthology.orchestra.run.trial125.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:48.645 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:48.645 INFO:teuthology.orchestra.run.trial125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:48.645 INFO:teuthology.orchestra.run.trial125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:48.645 INFO:teuthology.orchestra.run.trial125.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:48.646 INFO:teuthology.orchestra.run.trial125.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:48.646 INFO:teuthology.orchestra.run.trial125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:48.646 INFO:teuthology.orchestra.run.trial125.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:48.646 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:48.646 INFO:teuthology.orchestra.run.trial125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:49.203 INFO:teuthology.orchestra.run.trial125.stdout:Discarding blocks...Done. 2026-02-22T21:34:49.205 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial125.front.sepia.ceph.com -o noatime 2026-02-22T21:34:49.205 DEBUG:teuthology.orchestra.run.trial125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2026-02-22T21:34:49.273 DEBUG:teuthology.orchestra.run.trial125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2026-02-22T21:34:49.340 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2026-02-22T21:34:49.404 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2026-02-22T21:34:49.404 INFO:tasks.ceph:role: osd.14 2026-02-22T21:34:49.404 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:34:49.404 DEBUG:teuthology.orchestra.run.trial125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-22T21:34:49.468 INFO:teuthology.orchestra.run.trial125.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:34:49.468 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:34:49.468 INFO:teuthology.orchestra.run.trial125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:34:49.468 INFO:teuthology.orchestra.run.trial125.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:34:49.469 INFO:teuthology.orchestra.run.trial125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:34:50.026 INFO:teuthology.orchestra.run.trial125.stdout:Discarding blocks...Done. 2026-02-22T21:34:50.027 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial125.front.sepia.ceph.com -o noatime 2026-02-22T21:34:50.027 DEBUG:teuthology.orchestra.run.trial125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2026-02-22T21:34:50.097 DEBUG:teuthology.orchestra.run.trial125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2026-02-22T21:34:50.163 DEBUG:teuthology.orchestra.run.trial125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:50.241 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:50.239+0000 7fd37c810780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-22T21:34:50.241 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:50.239+0000 7fd37c810780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-22T21:34:50.242 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:50.239+0000 7fd37c810780 -1 bdev(0x555c20769800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:50.242 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:50.240+0000 7fd37c810780 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-22T21:34:54.128 DEBUG:teuthology.orchestra.run.trial125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-22T21:34:54.195 DEBUG:teuthology.orchestra.run.trial125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:54.273 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:54.271+0000 7fd70a334780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2026-02-22T21:34:54.273 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:54.271+0000 7fd70a334780 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2026-02-22T21:34:54.273 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:54.272+0000 7fd70a334780 -1 bdev(0x55b1cdceb800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:54.273 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:54.272+0000 7fd70a334780 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2026-02-22T21:34:58.154 DEBUG:teuthology.orchestra.run.trial125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2026-02-22T21:34:58.219 DEBUG:teuthology.orchestra.run.trial125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:34:58.297 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:58.295+0000 7f5a3b3b2780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2026-02-22T21:34:58.297 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:58.295+0000 7f5a3b3b2780 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2026-02-22T21:34:58.297 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:58.295+0000 7f5a3b3b2780 -1 bdev(0x564aab357800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2026-02-22T21:34:58.297 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:34:58.296+0000 7f5a3b3b2780 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2026-02-22T21:35:02.183 DEBUG:teuthology.orchestra.run.trial125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2026-02-22T21:35:02.248 DEBUG:teuthology.orchestra.run.trial125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:02.326 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:35:02.324+0000 7f5ebdcfe780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2026-02-22T21:35:02.327 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:35:02.324+0000 7f5ebdcfe780 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2026-02-22T21:35:02.327 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:35:02.324+0000 7f5ebdcfe780 -1 bdev(0x55b291b6f800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2026-02-22T21:35:02.327 INFO:teuthology.orchestra.run.trial125.stderr:2026-02-22T21:35:02.324+0000 7f5ebdcfe780 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2026-02-22T21:35:06.217 DEBUG:teuthology.orchestra.run.trial125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2026-02-22T21:35:06.281 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-22T21:35:06.307 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2026-02-22T21:35:06.307 INFO:tasks.ceph:role: osd.3 2026-02-22T21:35:06.307 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:35:06.307 DEBUG:teuthology.orchestra.run.trial140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T21:35:06.373 INFO:teuthology.orchestra.run.trial140.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:35:06.373 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:35:06.374 INFO:teuthology.orchestra.run.trial140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:35:06.941 INFO:teuthology.orchestra.run.trial140.stdout:Discarding blocks...Done. 2026-02-22T21:35:06.942 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial140.front.sepia.ceph.com -o noatime 2026-02-22T21:35:06.943 DEBUG:teuthology.orchestra.run.trial140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2026-02-22T21:35:07.011 DEBUG:teuthology.orchestra.run.trial140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-22T21:35:07.078 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2026-02-22T21:35:07.142 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2026-02-22T21:35:07.143 INFO:tasks.ceph:role: osd.7 2026-02-22T21:35:07.143 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:35:07.143 DEBUG:teuthology.orchestra.run.trial140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T21:35:07.209 INFO:teuthology.orchestra.run.trial140.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:35:07.210 INFO:teuthology.orchestra.run.trial140.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:35:07.211 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:35:07.211 INFO:teuthology.orchestra.run.trial140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:35:07.768 INFO:teuthology.orchestra.run.trial140.stdout:Discarding blocks...Done. 2026-02-22T21:35:07.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial140.front.sepia.ceph.com -o noatime 2026-02-22T21:35:07.771 DEBUG:teuthology.orchestra.run.trial140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2026-02-22T21:35:07.839 DEBUG:teuthology.orchestra.run.trial140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2026-02-22T21:35:07.906 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2026-02-22T21:35:07.970 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2026-02-22T21:35:07.971 INFO:tasks.ceph:role: osd.11 2026-02-22T21:35:07.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:35:07.971 DEBUG:teuthology.orchestra.run.trial140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T21:35:08.035 INFO:teuthology.orchestra.run.trial140.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:35:08.036 INFO:teuthology.orchestra.run.trial140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:35:08.593 INFO:teuthology.orchestra.run.trial140.stdout:Discarding blocks...Done. 2026-02-22T21:35:08.595 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial140.front.sepia.ceph.com -o noatime 2026-02-22T21:35:08.595 DEBUG:teuthology.orchestra.run.trial140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2026-02-22T21:35:08.664 DEBUG:teuthology.orchestra.run.trial140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2026-02-22T21:35:08.733 DEBUG:teuthology.orchestra.run.trial140:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2026-02-22T21:35:08.797 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2026-02-22T21:35:08.797 INFO:tasks.ceph:role: osd.15 2026-02-22T21:35:08.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:35:08.798 DEBUG:teuthology.orchestra.run.trial140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout: = sunit=0 swidth=0 blks 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T21:35:08.861 INFO:teuthology.orchestra.run.trial140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T21:35:09.419 INFO:teuthology.orchestra.run.trial140.stdout:Discarding blocks...Done. 2026-02-22T21:35:09.421 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial140.front.sepia.ceph.com -o noatime 2026-02-22T21:35:09.421 DEBUG:teuthology.orchestra.run.trial140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2026-02-22T21:35:09.491 DEBUG:teuthology.orchestra.run.trial140:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2026-02-22T21:35:09.558 DEBUG:teuthology.orchestra.run.trial140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:09.636 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:09.633+0000 7f657020d780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-22T21:35:09.637 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:09.634+0000 7f657020d780 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-22T21:35:09.637 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:09.634+0000 7f657020d780 -1 bdev(0x563cee37d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-22T21:35:09.637 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:09.634+0000 7f657020d780 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-22T21:35:13.531 DEBUG:teuthology.orchestra.run.trial140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-22T21:35:13.597 DEBUG:teuthology.orchestra.run.trial140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:13.675 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:13.672+0000 7fde84410780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2026-02-22T21:35:13.675 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:13.672+0000 7fde84410780 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2026-02-22T21:35:13.675 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:13.672+0000 7fde84410780 -1 bdev(0x56202bf81800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2026-02-22T21:35:13.675 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:13.672+0000 7fde84410780 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2026-02-22T21:35:17.571 DEBUG:teuthology.orchestra.run.trial140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2026-02-22T21:35:17.636 DEBUG:teuthology.orchestra.run.trial140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:17.715 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:17.712+0000 7f83cc02e780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2026-02-22T21:35:17.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:17.712+0000 7f83cc02e780 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2026-02-22T21:35:17.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:17.712+0000 7f83cc02e780 -1 bdev(0x5637ff1bf800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2026-02-22T21:35:17.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:17.713+0000 7f83cc02e780 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2026-02-22T21:35:21.602 DEBUG:teuthology.orchestra.run.trial140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2026-02-22T21:35:21.667 DEBUG:teuthology.orchestra.run.trial140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:21.744 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:21.740+0000 7f2348e1e780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2026-02-22T21:35:21.745 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:21.741+0000 7f2348e1e780 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2026-02-22T21:35:21.745 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:21.741+0000 7f2348e1e780 -1 bdev(0x5624f4515800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2026-02-22T21:35:21.745 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:21.741+0000 7f2348e1e780 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2026-02-22T21:35:25.627 DEBUG:teuthology.orchestra.run.trial140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2026-02-22T21:35:25.692 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T21:35:25.693 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:25.693 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2026-02-22T21:35:25.719 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:25.719 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-22T21:35:25.782 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:25.782 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2026-02-22T21:35:25.845 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:25.845 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2026-02-22T21:35:25.908 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:25.908 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2026-02-22T21:35:25.972 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:25.972 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-22T21:35:25.998 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:25.998 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2026-02-22T21:35:26.061 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:26.061 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2026-02-22T21:35:26.124 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:26.124 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2026-02-22T21:35:26.188 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:26.188 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-22T21:35:26.214 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:26.214 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2026-02-22T21:35:26.277 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:26.277 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2026-02-22T21:35:26.340 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:26.340 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2026-02-22T21:35:26.403 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.403 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-22T21:35:26.428 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.428 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-22T21:35:26.491 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.491 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2026-02-22T21:35:26.555 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.555 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2026-02-22T21:35:26.619 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.619 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2026-02-22T21:35:26.683 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:26.683 DEBUG:teuthology.orchestra.run.trial140:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-22T21:35:26.739 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T21:35:26.739 DEBUG:teuthology.orchestra.run.trial051:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T21:35:26.742 DEBUG:teuthology.orchestra.run.trial068:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T21:35:26.745 DEBUG:teuthology.orchestra.run.trial125:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[mgr.y] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBPdptpeFaqNBAABvmYT7mojo+/iv9xWwQ+wA== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.0] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBTdptpVhRGGRAABv4HIPURpWQSKXVTDutJqA== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.4] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBXdptpJZttHBAAd3pXXukgZEc/9a5FBDzlgw== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.8] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBbdptppvkfHhAAZu9IjT+zbbtdAajM+yXzUA== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.12] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBfdptpHjNyHxAAJUSSo+tvBgqGz/170Enw6w== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.1] 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBmdptp/BMkMhAAydO0irAG3tUe+sF0gleXOw== 2026-02-22T21:35:26.762 INFO:teuthology.orchestra.run.trial051.stdout:[osd.5] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBqdptpq83CMxAAcnPUXozabzMuIH6NPiI3Fg== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.9] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBudptp/kYeNhAA1MdpljIlwc2T1KM8ZHDueA== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.13] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBydptpxkxLNxAALClV3+NW13fFl7G6fiTAKQ== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.2] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQB6dptpmYVUDhAAj17ftr2FYQsI/8GXPS/wJQ== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.6] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQB+dptpK7o+EBAACouTSffVsxbTi4xi+4NQ5w== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.10] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCCdptpI7mrERAA4k+bHQJ7klFZfSQpKL/G1A== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.14] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCGdptpftphExAALFq7x42UlnZyAuAWbWaMgw== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[mgr.x] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBPdptpCC1jNxAAdqQ4Uz6/a/EzQHmPvcGevg== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.3] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCNdptp2ijQJRAADnZNtC94PSczQd1ZO6o5pw== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.7] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCRdptpId4VKBAAlRPiSpeodUBV5Aa3JxaDhQ== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.11] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCVdptpkPmAKhAAFOL0hY43UXtKpbtBS5t7gg== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[osd.15] 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout: key = AQCZdptpZEJALBAAs87OEZMmIiReS1Qb8YaVsg== 2026-02-22T21:35:26.763 INFO:teuthology.orchestra.run.trial051.stdout:[client.0] 2026-02-22T21:35:26.764 INFO:teuthology.orchestra.run.trial051.stdout: key = AQBQdptpnWkWARAA3PGW5nCvS9zVuWA7pmX3Nw== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[mgr.y] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBPdptpeFaqNBAABvmYT7mojo+/iv9xWwQ+wA== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.0] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBTdptpVhRGGRAABv4HIPURpWQSKXVTDutJqA== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.4] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBXdptpJZttHBAAd3pXXukgZEc/9a5FBDzlgw== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.8] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBbdptppvkfHhAAZu9IjT+zbbtdAajM+yXzUA== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.12] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBfdptpHjNyHxAAJUSSo+tvBgqGz/170Enw6w== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.1] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBmdptp/BMkMhAAydO0irAG3tUe+sF0gleXOw== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.5] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBqdptpq83CMxAAcnPUXozabzMuIH6NPiI3Fg== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.9] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBudptp/kYeNhAA1MdpljIlwc2T1KM8ZHDueA== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.13] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBydptpxkxLNxAALClV3+NW13fFl7G6fiTAKQ== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.2] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQB6dptpmYVUDhAAj17ftr2FYQsI/8GXPS/wJQ== 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout:[osd.6] 2026-02-22T21:35:26.766 INFO:teuthology.orchestra.run.trial068.stdout: key = AQB+dptpK7o+EBAACouTSffVsxbTi4xi+4NQ5w== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.10] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCCdptpI7mrERAA4k+bHQJ7klFZfSQpKL/G1A== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.14] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCGdptpftphExAALFq7x42UlnZyAuAWbWaMgw== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[mgr.x] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBPdptpCC1jNxAAdqQ4Uz6/a/EzQHmPvcGevg== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.3] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCNdptp2ijQJRAADnZNtC94PSczQd1ZO6o5pw== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.7] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCRdptpId4VKBAAlRPiSpeodUBV5Aa3JxaDhQ== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.11] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCVdptpkPmAKhAAFOL0hY43UXtKpbtBS5t7gg== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[osd.15] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQCZdptpZEJALBAAs87OEZMmIiReS1Qb8YaVsg== 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout:[client.0] 2026-02-22T21:35:26.767 INFO:teuthology.orchestra.run.trial068.stdout: key = AQBQdptpnWkWARAA3PGW5nCvS9zVuWA7pmX3Nw== 2026-02-22T21:35:26.768 INFO:teuthology.orchestra.run.trial125.stdout:[mgr.y] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBPdptpeFaqNBAABvmYT7mojo+/iv9xWwQ+wA== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.0] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBTdptpVhRGGRAABv4HIPURpWQSKXVTDutJqA== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.4] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBXdptpJZttHBAAd3pXXukgZEc/9a5FBDzlgw== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.8] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBbdptppvkfHhAAZu9IjT+zbbtdAajM+yXzUA== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.12] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBfdptpHjNyHxAAJUSSo+tvBgqGz/170Enw6w== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.1] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBmdptp/BMkMhAAydO0irAG3tUe+sF0gleXOw== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.5] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBqdptpq83CMxAAcnPUXozabzMuIH6NPiI3Fg== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.9] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBudptp/kYeNhAA1MdpljIlwc2T1KM8ZHDueA== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.13] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBydptpxkxLNxAALClV3+NW13fFl7G6fiTAKQ== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.2] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQB6dptpmYVUDhAAj17ftr2FYQsI/8GXPS/wJQ== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.6] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQB+dptpK7o+EBAACouTSffVsxbTi4xi+4NQ5w== 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout:[osd.10] 2026-02-22T21:35:26.769 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCCdptpI7mrERAA4k+bHQJ7klFZfSQpKL/G1A== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[osd.14] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCGdptpftphExAALFq7x42UlnZyAuAWbWaMgw== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[mgr.x] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBPdptpCC1jNxAAdqQ4Uz6/a/EzQHmPvcGevg== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[osd.3] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCNdptp2ijQJRAADnZNtC94PSczQd1ZO6o5pw== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[osd.7] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCRdptpId4VKBAAlRPiSpeodUBV5Aa3JxaDhQ== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[osd.11] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCVdptpkPmAKhAAFOL0hY43UXtKpbtBS5t7gg== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[osd.15] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQCZdptpZEJALBAAs87OEZMmIiReS1Qb8YaVsg== 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout:[client.0] 2026-02-22T21:35:26.770 INFO:teuthology.orchestra.run.trial125.stdout: key = AQBQdptpnWkWARAA3PGW5nCvS9zVuWA7pmX3Nw== 2026-02-22T21:35:26.771 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:26.806 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:26.810 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:26.851 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:26.886 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:26.889 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:26.932 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:26.967 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:26.970 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.011 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.048 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.051 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.090 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.130 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.132 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.172 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.210 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.213 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.253 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.291 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.293 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.332 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.371 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.374 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.413 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.449 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.453 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.491 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.529 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.532 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.572 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.608 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.611 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.651 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.689 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.692 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.732 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.769 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.771 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.812 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:27.849 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:27.852 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T21:35:27.891 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.929 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.931 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:27.971 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.008 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.011 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.050 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.088 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.091 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.133 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.170 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.173 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T21:35:28.211 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T21:35:28.249 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T21:35:28.252 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-22T21:35:28.293 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T21:35:28.293 DEBUG:teuthology.orchestra.run.trial051:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-22T21:35:28.353 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T21:35:28.439 DEBUG:teuthology.orchestra.run.trial051:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-22T21:35:28.464 DEBUG:teuthology.orchestra.run.trial068:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-22T21:35:28.488 DEBUG:teuthology.orchestra.run.trial068:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T21:35:28.577 DEBUG:teuthology.orchestra.run.trial068:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-22T21:35:28.601 DEBUG:teuthology.orchestra.run.trial125:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-22T21:35:28.626 DEBUG:teuthology.orchestra.run.trial125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T21:35:28.713 DEBUG:teuthology.orchestra.run.trial125:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-22T21:35:28.737 DEBUG:teuthology.orchestra.run.trial051:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:28.740 DEBUG:teuthology.orchestra.run.trial068:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:28.742 DEBUG:teuthology.orchestra.run.trial125:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T21:35:28.792 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-22T21:35:28.792 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2026-02-22T21:35:28.792 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-22T21:35:28.793 DEBUG:teuthology.orchestra.run.trial051:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2026-02-22T21:35:28.796 INFO:tasks.ceph.mon.a:Started 2026-02-22T21:35:28.796 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2026-02-22T21:35:28.796 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-22T21:35:28.796 DEBUG:teuthology.orchestra.run.trial068:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2026-02-22T21:35:28.799 INFO:tasks.ceph.mon.b:Started 2026-02-22T21:35:28.800 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2026-02-22T21:35:28.800 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-22T21:35:28.800 DEBUG:teuthology.orchestra.run.trial125:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2026-02-22T21:35:28.834 INFO:tasks.ceph.mon.c:Started 2026-02-22T21:35:28.834 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-22T21:35:28.834 INFO:tasks.ceph.mgr.y:Restarting daemon 2026-02-22T21:35:28.834 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2026-02-22T21:35:28.835 INFO:tasks.ceph.mgr.y:Started 2026-02-22T21:35:28.836 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-22T21:35:28.836 DEBUG:teuthology.orchestra.run.trial140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2026-02-22T21:35:28.837 INFO:tasks.ceph.mgr.x:Started 2026-02-22T21:35:28.837 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T21:35:28.837 DEBUG:teuthology.orchestra.run.trial051:> sudo ceph --cluster ceph osd crush tunables default 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.900+0000 7f404a15c640 10 monclient: get_monmap_and_config 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.900+0000 7f404a15c640 10 monclient: build_initial_monmap 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.900+0000 7f404a15c640 10 monclient: monmap: 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:35:28.901660+0000 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:35:28.901660+0000 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:28.902 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 Processor -- start 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 -- start start 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient: init 2026-02-22T21:35:28.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient: _add_conns ranks=[3,2,4,1,0,5] 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-a con 0x7f4044150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4044150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-b con 0x7f404415c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415a920 con 0x7f4044150600 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f4044150220 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f404415b240 con 0x7f404415c710 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f404a15c640 10 monclient(hunting): authenticate will time out at 1004.481963s 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f40437fe640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 msgr2=0x7f40440c73a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.901+0000 7f40437fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:28.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.902+0000 7f4042ffd640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4044150600 legacy=0x7f404415c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:28.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.909+0000 7f4043fff640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 msgr2=0x7f404406fca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:28.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:28.909+0000 7f4043fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:29.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.102+0000 7f40437fe640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 msgr2=0x7f40440c73a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:29.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.102+0000 7f40437fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:29.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.103+0000 7f4042ffd640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4044150600 legacy=0x7f404415c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:29.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.110+0000 7f4043fff640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 msgr2=0x7f404406fca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:29.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.110+0000 7f4043fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:29.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.503+0000 7f40437fe640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 msgr2=0x7f40440c73a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:29.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.503+0000 7f40437fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:29.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.504+0000 7f4042ffd640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4044150600 legacy=0x7f404415c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:29.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.511+0000 7f4043fff640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 msgr2=0x7f404406fca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:29.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:29.511+0000 7f4043fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:30.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:30.304+0000 7f40437fe640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 msgr2=0x7f40440c73a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:30.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:30.304+0000 7f40437fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:30.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:30.305+0000 7f4042ffd640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4044150600 legacy=0x7f404415c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:30.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:30.312+0000 7f4043fff640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 msgr2=0x7f404406fca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:30.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:30.312+0000 7f4043fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:31.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:31.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:31.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 msgr2=0x7f404406fca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f404415c710 0x7f404406fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 msgr2=0x7f40440c73a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4044150220 0x7f40440c73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.901+0000 7f40417fa640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4044150600 legacy=0x7f404415c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient: _add_conns ranks=[1,5,0] 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f40200037c0 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f40200037c0 legacy=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c con 0x7f4020007440 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4020009fe0 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415b520 con 0x7f4020009fe0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415a920 con 0x7f40200037c0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f4020007440 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f4020009fe0 legacy=0x7f402000c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:31.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:31.902+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:32.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.103+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f4020009fe0 legacy=0x7f402000c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:32.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.103+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:32.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.103+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:32.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.103+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f40200037c0 legacy=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:32.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.504+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f4020009fe0 legacy=0x7f402000c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:32.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.504+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:32.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.504+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f40200037c0 legacy=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:32.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:32.504+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:33.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:33.305+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f4020009fe0 legacy=0x7f402000c4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:33.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:33.305+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:33.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:33.305+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:33.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:33.305+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f40200037c0 legacy=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:34.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4020007440 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f40200037c0 legacy=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f4020009fe0 legacy=0x7f402000c4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient: _add_conns ranks=[2,4,3,1,5,0] 2026-02-22T21:35:34.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f402000edf0 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b con 0x7f402000f980 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a con 0x7f402000c5d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402000fe80 con 0x7f402000edf0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f402000c5d0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f404415b240 con 0x7f402000f980 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 msgr2=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:34.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:34.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:34.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f402000edf0 legacy=0x7f402000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:34.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:34.901+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:35.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.102+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 msgr2=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:35.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.102+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:35.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.102+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:35.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.102+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:35.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.102+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f402000edf0 legacy=0x7f402000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:35.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.503+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 msgr2=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:35.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.503+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:35.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.503+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:35.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.503+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:35.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:35.503+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f402000edf0 legacy=0x7f402000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:36.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:36.304+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 msgr2=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:36.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:36.304+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:36.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:36.304+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:36.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:36.304+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f402000edf0 legacy=0x7f402000f1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:36.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:36.304+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 msgr2=0x7f4020009900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f980 0x7f4020009900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 msgr2=0x7f4020005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f402000c5d0 0x7f4020005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f402000edf0 legacy=0x7f402000f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient: _add_conns ranks=[0,5,1] 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f402000dd60 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c con 0x7f402000e850 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f4020014710 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415b520 con 0x7f402000dd60 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402000fe80 con 0x7f4020014710 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f402000e850 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f40437fe640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000dd60 legacy=0x7f402000e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f4043fff640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f4020014710 legacy=0x7f4020016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 msgr2=0x7f40200141d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:37.901+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:38.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.102+0000 7f40437fe640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000dd60 legacy=0x7f402000e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:38.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.102+0000 7f4043fff640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f4020014710 legacy=0x7f4020016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:38.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.102+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 msgr2=0x7f40200141d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:38.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.102+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:38.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.503+0000 7f40437fe640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000dd60 legacy=0x7f402000e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:38.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.503+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 msgr2=0x7f40200141d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:38.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.503+0000 7f4043fff640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f4020014710 legacy=0x7f4020016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:38.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:38.503+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:39.290 INFO:tasks.ceph.mon.b.trial068.stderr:--00:00:00:10.442 44415-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/c8/2482ecb30cd94afc031e15355b32036f38cb43.debug 2026-02-22T21:35:39.295 INFO:tasks.ceph.mon.c.trial125.stderr:--00:00:00:10.411 44425-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/c8/2482ecb30cd94afc031e15355b32036f38cb43.debug 2026-02-22T21:35:39.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:39.304+0000 7f40437fe640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000dd60 legacy=0x7f402000e140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:39.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:39.304+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 msgr2=0x7f40200141d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:39.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:39.304+0000 7f4043fff640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f4020014710 legacy=0x7f4020016b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:39.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:39.304+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:39.331 INFO:tasks.ceph.mon.a.trial051.stderr:--00:00:00:10.485 44639-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/c8/2482ecb30cd94afc031e15355b32036f38cb43.debug 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 msgr2=0x7f40200141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000e850 0x7f40200141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f4020014710 legacy=0x7f4020016b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000dd60 legacy=0x7f402000e140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient: _add_conns ranks=[1,5,0,2,3,4] 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f402000c5d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c con 0x7f402000f500 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f402000e2a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415a920 con 0x7f402000e2a0 2026-02-22T21:35:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415b520 con 0x7f402000c5d0 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f402000f500 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000e2a0 legacy=0x7f4020006300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 msgr2=0x7f402000dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:40.901+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:41.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.102+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000e2a0 legacy=0x7f4020006300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:41.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.102+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 msgr2=0x7f402000dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:41.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.102+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:41.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.102+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.503+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000e2a0 legacy=0x7f4020006300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.503+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 msgr2=0x7f402000dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.503+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:41.503+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:42.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:42.304+0000 7f4043fff640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000e2a0 legacy=0x7f4020006300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.51:6789/0 2026-02-22T21:35:42.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:42.304+0000 7f4042ffd640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 msgr2=0x7f402000dd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.125:3300/0 2026-02-22T21:35:42.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:42.304+0000 7f40437fe640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.68:6789/0 2026-02-22T21:35:42.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:42.304+0000 7f4042ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:43.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 msgr2=0x7f402000dd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f402000f500 0x7f402000dd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- >> v1:10.20.193.68:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- >> v1:10.20.193.51:6789/0 conn(0x7f402000e2a0 legacy=0x7f4020006300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient: _add_conns ranks=[2,4,3] 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4020009900 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:43.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b con 0x7f4020005bf0 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4020013130 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402000fe80 con 0x7f4020009900 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f4020013130 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f404415b240 con 0x7f4020005bf0 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 msgr2=0x7f40200155d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:43.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4020009900 legacy=0x7f4020006a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:43.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:43.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 msgr2=0x7f402000e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:43.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:43.901+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:44.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.102+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 msgr2=0x7f40200155d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:44.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.102+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:44.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.102+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4020009900 legacy=0x7f4020006a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:44.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.102+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 msgr2=0x7f402000e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:44.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.102+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:44.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.503+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 msgr2=0x7f40200155d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:44.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.503+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:44.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.503+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 msgr2=0x7f402000e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:44.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.503+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4020009900 legacy=0x7f4020006a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:44.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:44.503+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2026-02-22T21:35:44.833 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:00:15.985 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:44.854 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:00:15.970 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:44.888 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:00:16.041 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:44.985 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:00:16.136 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:45.005 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:00:16.122 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:45.039 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:00:16.193 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:35:45.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:45.304+0000 7f4043fff640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 msgr2=0x7f40200155d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.51:3300/0 2026-02-22T21:35:45.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:45.304+0000 7f4043fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:45.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:45.304+0000 7f4042ffd640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 msgr2=0x7f402000e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:10.20.193.68:3300/0 2026-02-22T21:35:45.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:45.304+0000 7f40437fe640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4020009900 legacy=0x7f4020006a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:10.20.193.125:6789/0 2026-02-22T21:35:45.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:45.304+0000 7f4042ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2026-02-22T21:35:46.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 10 monclient(hunting): tick 2026-02-22T21:35:46.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 10 monclient(hunting): _check_auth_tickets 2026-02-22T21:35:46.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 monclient(hunting): continuing hunt 2026-02-22T21:35:46.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 10 monclient(hunting): _reopen_session rank -1 2026-02-22T21:35:46.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 -- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 msgr2=0x7f402000e6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4020005bf0 0x7f402000e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 -- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 msgr2=0x7f40200155d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4020013130 0x7f40200155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 1 -- >> v1:10.20.193.125:6789/0 conn(0x7f4020009900 legacy=0x7f4020006a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f4043fff640 1 -- reap_dead start 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 10 monclient: _add_conns ranks=[0,2,4,1,5,3] 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.901+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f402000c5d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f402000e6d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 10 monclient(hunting): picked mon.noname-b con 0x7f402000f500 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:46.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404415a920 con 0x7f402000c5d0 2026-02-22T21:35:46.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402000fe80 con 0x7f402000e6d0 2026-02-22T21:35:46.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:46.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f402000f500 2026-02-22T21:35:46.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.902+0000 7f40417fa640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:46.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.916+0000 7f4043fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:46.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.928+0000 7f40437fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f402000c5d0 0x7f402000dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:33484/0 (socket says 10.20.193.51:33484) 2026-02-22T21:35:46.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.928+0000 7f40437fe640 1 -- 10.20.193.51:0/4178468879 learned_addr learned my addr 10.20.193.51:0/4178468879 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:35:46.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.939+0000 7f4043fff640 10 monclient(hunting): get_auth_request con 0x7f402000f500 auth_method 0 2026-02-22T21:35:46.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.939+0000 7f4043fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:46.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.939+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:46.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.939+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:46.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.941+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 msgr2=0x7f402000cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2026-02-22T21:35:46.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.941+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 msgr2=0x7f402000cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2026-02-22T21:35:46.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.941+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:35:46.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:46.941+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:47.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.171+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.171+0000 7f4043fff640 10 monclient(hunting): get_auth_request con 0x7f402000f500 auth_method 0 2026-02-22T21:35:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.171+0000 7f4043fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.171+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.171+0000 7f4043fff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.173+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 msgr2=0x7f402000cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2026-02-22T21:35:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.173+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 msgr2=0x7f402000cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2026-02-22T21:35:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.173+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:35:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.173+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2026-02-22T21:35:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1241737826 0 0) 0x7f402000fe80 con 0x7f402000e6d0 2026-02-22T21:35:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 10 monclient(hunting): _negotiate 2026-02-22T21:35:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:47.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 10 monclient(hunting): my global_id is 4100 2026-02-22T21:35:47.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.262+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4028003620 con 0x7f402000e6d0 2026-02-22T21:35:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923024058 0 0) 0x7f404415a920 con 0x7f402000c5d0 2026-02-22T21:35:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 10 monclient(hunting): _negotiate 2026-02-22T21:35:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 10 monclient(hunting): my global_id is 4098 2026-02-22T21:35:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.268+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f402000fe80 con 0x7f402000c5d0 2026-02-22T21:35:47.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 259226683 0 0) 0x7f4028003620 con 0x7f402000e6d0 2026-02-22T21:35:47.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 msgr2=0x7f402000cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:47.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 --2- 10.20.193.51:0/4178468879 >> v2:10.20.193.68:3300/0 conn(0x7f402000f500 0x7f402000cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 >> v1:10.20.193.51:6789/0 conn(0x7f402000c5d0 legacy=0x7f402000dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404415b520 con 0x7f402000e6d0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f4034004160 con 0x7f402000e6d0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:35:47.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: dump: 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:35:47.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 >> v1:10.20.193.125:6789/0 conn(0x7f402000e6d0 legacy=0x7f402000edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4028003c90 0x7f4028004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): picked mon.b con 0x7f4028003c90 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028004900 0x7f4028004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): picked mon.c con 0x7f4028004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 0x7f402800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): picked mon.a con 0x7f4028005430 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f404415af60 con 0x7f4028005430 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f402800c380 con 0x7f4028003c90 2026-02-22T21:35:47.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f402800c580 con 0x7f4028004900 2026-02-22T21:35:47.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:47.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.286+0000 7f40427fc640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:35:47.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.302+0000 7f40437fe640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4028003c90 0x7f4028004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:47.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.307+0000 7f40437fe640 10 monclient(hunting): get_auth_request con 0x7f4028003c90 auth_method 0 2026-02-22T21:35:47.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.307+0000 7f40437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:47.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.307+0000 7f40437fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.307+0000 7f40437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:47.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.309+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 0x7f402800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:47.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.310+0000 7f4043fff640 10 monclient(hunting): get_auth_request con 0x7f4028005430 auth_method 0 2026-02-22T21:35:47.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.310+0000 7f4043fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:47.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.310+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.310+0000 7f4043fff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f4043fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f4043fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f4043fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f40437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f40437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:47.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.311+0000 7f40437fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:35:47.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.312+0000 7f4042ffd640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028004900 0x7f4028004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:47.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.314+0000 7f4042ffd640 10 monclient(hunting): get_auth_request con 0x7f4028004900 auth_method 0 2026-02-22T21:35:47.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.314+0000 7f4042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:47.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.314+0000 7f4042ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.314+0000 7f4042ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:47.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 10 monclient(hunting): handle_auth_done global_id 4100 payload 293 2026-02-22T21:35:47.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028004900 msgr2=0x7f4028004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:47.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028004900 0x7f4028004d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4028003c90 msgr2=0x7f4028004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4028003c90 0x7f4028004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 10 monclient: _finish_hunting 0 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 monclient: found mon.a 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4028003620 con 0x7f4028005430 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 10 monclient: _finish_auth 0 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 10 monclient: _check_auth_tickets 2026-02-22T21:35:47.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:47.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f404a15c640 5 monclient: authenticate success, global_id 4100 2026-02-22T21:35:47.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f404a15c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:47.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.321+0000 7f4043fff640 1 -- 10.20.193.51:0/4178468879 reap_dead start 2026-02-22T21:35:47.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.326+0000 7f4043fff640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 0x7f402800bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f40440c62e0 tx=0x7f403802ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8980a5a113f39c20 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:47.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403803d070 con 0x7f4028005430 2026-02-22T21:35:47.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:47.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: dump: 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:47.357 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40380309e0 con 0x7f4028005430 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 1 -- 10.20.193.51:0/4178468879 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4038007b30 con 0x7f4028005430 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 4 set_mon_vals no callback set 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f40427fc640 10 monclient: dump: 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:47.358 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 10 monclient: shutdown 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 1 -- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 msgr2=0x7f402800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 0x7f402800bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f40440c62e0 tx=0x7f403802ffa0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 1 -- 10.20.193.51:0/4178468879 shutdown_connections 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.351+0000 7f404a15c640 1 --2- 10.20.193.51:0/4178468879 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4028005430 0x7f402800bac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 -- 10.20.193.51:0/4178468879 >> 10.20.193.51:0/4178468879 conn(0x7f40440cd7b0 msgr2=0x7f40440cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 -- 10.20.193.51:0/4178468879 shutdown_connections 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 -- 10.20.193.51:0/4178468879 wait complete. 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 10 monclient: build_initial_monmap 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 10 monclient: monmap: 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:47.360 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 librados: starting msgr at 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 librados: starting objecter 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 Processor -- start 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 -- start start 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 librados: setting wanted keys 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 1 librados: calling monclient init 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.352+0000 7f404a15c640 10 monclient: init 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:47.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028005430 0x7f40440c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-c con 0x7f4028005430 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-b con 0x7f40440c1420 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40440739f0 0x7f4044073e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): picked mon.noname-a con 0x7f40440739f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f40440c73a0 con 0x7f40440739f0 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f404415bae0 con 0x7f40440c1420 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:47.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f40440c7520 con 0x7f4028005430 2026-02-22T21:35:47.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:47.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.353+0000 7f404a15c640 10 monclient(hunting): authenticate will time out at 1022.934356s 2026-02-22T21:35:47.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:33474/0 (socket says 10.20.193.51:33474) 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 1 -- 10.20.193.51:0/3288764786 learned_addr learned my addr 10.20.193.51:0/3288764786 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): get_auth_request con 0x7f40440c1420 auth_method 0 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:47.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.356+0000 7f4042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 -- 10.20.193.51:0/3288764786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028005430 msgr2=0x7f40440c3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028005430 0x7f40440c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 -- 10.20.193.51:0/3288764786 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40440739f0 msgr2=0x7f4044073e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40440739f0 0x7f4044073e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 monclient: found mon.noname-b 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:47.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:47.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.358+0000 7f4042ffd640 1 -- 10.20.193.51:0/3288764786 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40440c1de0 con 0x7f40440c1420 2026-02-22T21:35:47.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.359+0000 7f4042ffd640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4034008a30 tx=0x7f4034008b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:47.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.359+0000 7f401ffff640 10 client.?.objecter ms_handle_connect 0x7f40440c1420 2026-02-22T21:35:47.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.359+0000 7f401ffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:47.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403400e8b0 con 0x7f40440c1420 2026-02-22T21:35:47.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:47.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: dump: 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:47.370 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: _finish_auth 0 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403400ea50 con 0x7f40440c1420 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403400ee40 con 0x7f40440c1420 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f401ffff640 10 monclient: dump: 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:47.371 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f4041ffb640 4 set_mon_vals no callback set 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 5 monclient: authenticate success, global_id 4108 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 10 monclient: _renew_subs 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:47.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40440743b0 con 0x7f40440c1420 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 10 monclient: _renew_subs 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404406d350 con 0x7f40440c1420 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.364+0000 7f404a15c640 1 librados: init done 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.365+0000 7f404a15c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.365+0000 7f404a15c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.365+0000 7f404a15c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:47.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.365+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f404415c000 con 0x7f40440c1420 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f4034013030 con 0x7f40440c1420 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4034014560 con 0x7f40440c1420 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 10 client.4108.objecter ms_dispatch2 0x7f4044000b90 osd_map(1..1 src has 1..1) 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2026-02-22T21:35:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.376+0000 7f401ffff640 20 client.4108.objecter dump_active .. 0 homeless 2026-02-22T21:35:47.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.450+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f40440c1de0 con 0x7f40440c1420 2026-02-22T21:35:47.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.450+0000 7f401ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:47.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.450+0000 7f401ffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:47.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.480+0000 7f404a15c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2026-02-22T21:35:47.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.480+0000 7f404a15c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2026-02-22T21:35:47.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.480+0000 7f404a15c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:47.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.480+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4044074da0 con 0x7f40440c1420 2026-02-22T21:35:47.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.615+0000 7f401ffff640 1 -- 10.20.193.51:0/3288764786 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f4034011070 con 0x7f40440c1420 2026-02-22T21:35:47.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.615+0000 7f401ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2026-02-22T21:35:47.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.615+0000 7f401ffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2026-02-22T21:35:47.619 INFO:teuthology.orchestra.run.trial051.stderr:adjusted tunables profile to default 2026-02-22T21:35:47.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 10 librados: watch_flush enter 2026-02-22T21:35:47.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 10 librados: watch_flush exit 2026-02-22T21:35:47.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 20 client.4108.objecter shutdown clearing up homeless session... 2026-02-22T21:35:47.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 10 client.4108.objecter successfully canceled tick 2026-02-22T21:35:47.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 10 monclient: shutdown 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 msgr2=0x7f40440c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4034008a30 tx=0x7f4034008b30 comp rx=0 tx=0).stop 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 shutdown_connections 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40440739f0 0x7f4044073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f40440c1420 0x7f40440c18a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 --2- 10.20.193.51:0/3288764786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4028005430 0x7f40440c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 >> 10.20.193.51:0/3288764786 conn(0x7f40440cd7b0 msgr2=0x7f40440cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 shutdown_connections 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 -- 10.20.193.51:0/3288764786 wait complete. 2026-02-22T21:35:47.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:47.616+0000 7f404a15c640 1 librados: shutdown 2026-02-22T21:35:47.627 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T21:35:47.627 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-22T21:35:47.628 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:47.628 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-22T21:35:47.651 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:47.652 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2026-02-22T21:35:47.715 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:47.715 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2026-02-22T21:35:47.725 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.722+0000 7f62ac9ce140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T21:35:47.743 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.739+0000 7f4d79bcc140 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T21:35:47.747 DEBUG:teuthology.orchestra.run.trial051:> set -ex 2026-02-22T21:35:47.747 DEBUG:teuthology.orchestra.run.trial051:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2026-02-22T21:35:47.811 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:47.811 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-22T21:35:47.838 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:47.838 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2026-02-22T21:35:47.846 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.843+0000 7f62ac9ce140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T21:35:47.864 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.860+0000 7f4d79bcc140 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T21:35:47.872 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.870+0000 7f62ac9ce140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T21:35:47.891 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.887+0000 7f4d79bcc140 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T21:35:47.903 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:47.903 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2026-02-22T21:35:47.905 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.903+0000 7f62ac9ce140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T21:35:47.924 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.920+0000 7f4d79bcc140 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T21:35:47.928 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.926+0000 7f62ac9ce140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T21:35:47.947 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.943+0000 7f4d79bcc140 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T21:35:47.954 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:47.952+0000 7f62ac9ce140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T21:35:47.967 DEBUG:teuthology.orchestra.run.trial068:> set -ex 2026-02-22T21:35:47.967 DEBUG:teuthology.orchestra.run.trial068:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2026-02-22T21:35:47.973 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:47.969+0000 7f4d79bcc140 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T21:35:48.010 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.007+0000 7f62ac9ce140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T21:35:48.028 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.024+0000 7f4d79bcc140 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T21:35:48.031 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:48.031 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-22T21:35:48.059 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:48.059 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2026-02-22T21:35:48.070 INFO:tasks.ceph.mgr.y.trial051.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T21:35:48.070 INFO:tasks.ceph.mgr.y.trial051.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T21:35:48.070 INFO:tasks.ceph.mgr.y.trial051.stderr: from numpy import show_config as show_numpy_config 2026-02-22T21:35:48.073 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.071+0000 7f62ac9ce140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T21:35:48.089 INFO:tasks.ceph.mgr.x.trial140.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T21:35:48.089 INFO:tasks.ceph.mgr.x.trial140.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T21:35:48.089 INFO:tasks.ceph.mgr.x.trial140.stderr: from numpy import show_config as show_numpy_config 2026-02-22T21:35:48.092 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.088+0000 7f4d79bcc140 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T21:35:48.099 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.096+0000 7f62ac9ce140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T21:35:48.117 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.113+0000 7f4d79bcc140 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T21:35:48.125 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:48.125 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2026-02-22T21:35:48.154 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.151+0000 7f62ac9ce140 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2026-02-22T21:35:48.172 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.168+0000 7f4d79bcc140 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2026-02-22T21:35:48.189 DEBUG:teuthology.orchestra.run.trial125:> set -ex 2026-02-22T21:35:48.189 DEBUG:teuthology.orchestra.run.trial125:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2026-02-22T21:35:48.196 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.194+0000 7f62ac9ce140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T21:35:48.215 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.211+0000 7f4d79bcc140 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T21:35:48.222 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.219+0000 7f62ac9ce140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T21:35:48.240 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.236+0000 7f4d79bcc140 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T21:35:48.247 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.244+0000 7f62ac9ce140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T21:35:48.253 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:48.253 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-22T21:35:48.265 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.261+0000 7f4d79bcc140 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T21:35:48.272 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.269+0000 7f62ac9ce140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T21:35:48.279 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:48.279 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2026-02-22T21:35:48.290 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.286+0000 7f4d79bcc140 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T21:35:48.294 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.292+0000 7f62ac9ce140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T21:35:48.312 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.309+0000 7f4d79bcc140 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T21:35:48.314 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:48.314 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2026-02-22T21:35:48.377 DEBUG:teuthology.orchestra.run.trial140:> set -ex 2026-02-22T21:35:48.378 DEBUG:teuthology.orchestra.run.trial140:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2026-02-22T21:35:48.443 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new b8bc99a9-1c6a-4062-808d-e1fe0423f69b 0 2026-02-22T21:35:48.486 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.484+0000 7f62ac9ce140 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2026-02-22T21:35:48.506 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.502+0000 7f4d79bcc140 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2026-02-22T21:35:48.541 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.538+0000 7f62ac9ce140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T21:35:48.547 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 10 monclient: get_monmap_and_config 2026-02-22T21:35:48.547 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 10 monclient: build_initial_monmap 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 10 monclient: monmap: 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:48.544897+0000 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:48.544897+0000 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:48.548 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:48.549 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.552 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 1 Processor -- start 2026-02-22T21:35:48.552 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 1 -- start start 2026-02-22T21:35:48.552 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 10 monclient: init 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.543+0000 7f221ce85640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient: _add_conns ranks=[2,1,4,5,0,3] 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2218114a50 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f221811d670 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f2218121210 0x7f22181236d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-b con 0x7f2218121210 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2218123df0 con 0x7f221811d670 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2218125010 con 0x7f2218114a50 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f2218126210 con 0x7f2218121210 2026-02-22T21:35:48.553 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f221ce85640 10 monclient(hunting): authenticate will time out at 1021.171071s 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f2216575640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f2218114a50 0x7f2218114e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.140:34504/0 (socket says 10.20.193.140:34504) 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f2216575640 1 -- 10.20.193.140:0/532659759 learned_addr learned my addr 10.20.193.140:0/532659759 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.544+0000 7f2216d76640 1 --2- 10.20.193.140:0/532659759 >> v2:10.20.193.68:3300/0 conn(0x7f2218121210 0x7f22181236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): get_auth_request con 0x7f2218121210 auth_method 0 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.545+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.554 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 1 -- 10.20.193.140:0/532659759 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791384066 0 0) 0x7f2218125010 con 0x7f2218114a50 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 10 monclient(hunting): _negotiate 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 10 monclient(hunting): my global_id is 4109 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2215573640 1 -- 10.20.193.140:0/532659759 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2204003620 con 0x7f2218114a50 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 1 -- 10.20.193.140:0/532659759 >> v1:10.20.193.125:6789/0 conn(0x7f2218114a50 legacy=0x7f2218114e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 1 -- 10.20.193.140:0/532659759 >> v1:10.20.193.68:6789/0 conn(0x7f221811d670 legacy=0x7f221811fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 10 monclient: _finish_hunting 0 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 1 monclient: found mon.noname-b 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:48.555 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.556 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.546+0000 7f2216d76640 1 -- 10.20.193.140:0/532659759 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2218126390 con 0x7f2218121210 2026-02-22T21:35:48.563 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.561+0000 7f62ac9ce140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.549+0000 7f2216d76640 1 --2- 10.20.193.140:0/532659759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218121210 0x7f22181236d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2208009d90 tx=0x7f220802f230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8559f9079a2fac server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 1 -- 10.20.193.140:0/532659759 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220803c070 con 0x7f2218121210 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: dump: 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.565 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: _finish_auth 0 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: _check_auth_tickets 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 1 -- 10.20.193.140:0/532659759 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2208037440 con 0x7f2218121210 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:48.566 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 5 monclient: authenticate success, global_id 4111 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 1 -- 10.20.193.140:0/532659759 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208037810 con 0x7f2218121210 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 4 set_mon_vals no callback set 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f2215573640 10 monclient: dump: 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.567 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 10 monclient: shutdown 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 -- 10.20.193.140:0/532659759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218121210 msgr2=0x7f22181236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 --2- 10.20.193.140:0/532659759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218121210 0x7f22181236d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2208009d90 tx=0x7f220802f230 comp rx=0 tx=0).stop 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 -- 10.20.193.140:0/532659759 shutdown_connections 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 --2- 10.20.193.140:0/532659759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218121210 0x7f22181236d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 -- 10.20.193.140:0/532659759 >> 10.20.193.140:0/532659759 conn(0x7f2218073410 msgr2=0x7f2218073820 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 -- 10.20.193.140:0/532659759 shutdown_connections 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 -- 10.20.193.140:0/532659759 wait complete. 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 10 monclient: build_initial_monmap 2026-02-22T21:35:48.568 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 10 monclient: monmap: 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 librados: starting msgr at 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.551+0000 7f221ce85640 1 librados: starting objecter 2026-02-22T21:35:48.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 Processor -- start 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 -- start start 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 librados: setting wanted keys 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 librados: calling monclient init 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient: init 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-b con 0x7f2218114a50 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f2218118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-a con 0x7f221811d670 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:48.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f22181b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): picked mon.noname-c con 0x7f2218121210 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2218124440 con 0x7f221811d670 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f22181242c0 con 0x7f2218114a50 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f22181245c0 con 0x7f2218121210 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f221ce85640 10 monclient(hunting): authenticate will time out at 1021.179209s 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.571 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216d76640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f22181b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.140:46352/0 (socket says 10.20.193.140:46352) 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 learned_addr learned my addr 10.20.193.140:0/1042482878 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): get_auth_request con 0x7f2218114a50 auth_method 0 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2215d74640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f2218118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216d76640 10 monclient(hunting): get_auth_request con 0x7f2218121210 auth_method 0 2026-02-22T21:35:48.572 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.552+0000 7f2216d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 10 monclient(hunting): get_auth_request con 0x7f221811d670 auth_method 0 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 msgr2=0x7f2218118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 msgr2=0x7f2218118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f2218118f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2215d74640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f2218118f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.573 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.553+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 msgr2=0x7f22181b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f22181b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 msgr2=0x7f2218118f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f2218118f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 10 monclient: _finish_hunting 0 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 monclient: found mon.noname-b 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.574 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.554+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22181b1b70 con 0x7f2218114a50 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.555+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f220c00c700 tx=0x7f220c00cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.555+0000 7f21feffd640 10 client.?.objecter ms_handle_connect 0x7f2218114a50 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.555+0000 7f21feffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c017070 con 0x7f2218114a50 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:48.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: dump: 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.576 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: _finish_auth 0 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: _check_auth_tickets 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 5 monclient: authenticate success, global_id 4114 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _renew_subs 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.577 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22181b1e00 con 0x7f2218114a50 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _renew_subs 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22181b8510 con 0x7f2218114a50 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 1 librados: init done 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220c005bd0 con 0x7f2218114a50 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c007070 con 0x7f2218114a50 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.578 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f21feffd640 10 monclient: dump: 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f2214d72640 4 set_mon_vals no callback set 2026-02-22T21:35:48.579 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.556+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2218124aa0 con 0x7f2218114a50 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 msgr2=0x7f2218118910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 msgr2=0x7f2218118910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f220c00c700 tx=0x7f220c00cbd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f220c00c700 tx=0x7f220c00cbd0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient: ms_handle_reset current mon [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 msgr2=0x7f2218118910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2218114a50 0x7f2218118910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f21e4007300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.580 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): picked mon.a con 0x7f221811d670 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f21e4007cd0 0x7f21e400a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): picked mon.b con 0x7f21e4007cd0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f21e400cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): picked mon.c con 0x7f2218121210 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f21e400d690 con 0x7f221811d670 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f21e400d810 con 0x7f21e4007cd0 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f21e400d990 con 0x7f2218121210 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f21feffd640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216d76640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f21e400cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f21e4007300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216d76640 10 monclient(hunting): get_auth_request con 0x7f2218121210 auth_method 0 2026-02-22T21:35:48.581 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216d76640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 10 monclient(hunting): get_auth_request con 0x7f221811d670 auth_method 0 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.557+0000 7f2216575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.582 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:35:48.582 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.558+0000 7f4d79bcc140 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T21:35:48.583 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216575640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient(hunting): handle_auth_done global_id 4114 payload 293 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f21e4007cd0 msgr2=0x7f21e400a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f21e4007cd0 0x7f21e400a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 msgr2=0x7f21e4007300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f21e4007300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient: _finish_hunting 0 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 monclient: found mon.c 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f21e400dcc0 con 0x7f2218121210 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:48.584 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.558+0000 7f2216d76640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2218126390 con 0x7f2218121210 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f2216d76640 10 monclient: _finish_auth 0 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f2216d76640 10 monclient: _check_auth_tickets 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f2216d76640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f2216d76640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f21e400cc80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2208037680 tx=0x7f220c007e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f21feffd640 10 client.4114.objecter ms_handle_connect 0x7f2218121210 2026-02-22T21:35:48.585 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.559+0000 7f21feffd640 10 client.4114.objecter resend_mon_ops 2026-02-22T21:35:48.585 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.580+0000 7f4d79bcc140 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T21:35:48.587 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.584+0000 7f62ac9ce140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T21:35:48.607 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.604+0000 7f4d79bcc140 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208046030 con 0x7f2218121210 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 10 monclient: dump: 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220803d040 con 0x7f2218121210 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21feffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:48.635 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.631+0000 7f21fffff640 4 set_mon_vals no callback set 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f2208037ac0 con 0x7f2218121210 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 4 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f2208042730 con 0x7f2218121210 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 10 client.4114.objecter ms_dispatch2 0x7f2218000b90 osd_map(2..2 src has 1..2) 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 20 client.4114.objecter dump_active .. 0 homeless 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f22080493d0 con 0x7f2218121210 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.639 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.635+0000 7f21feffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:48.639 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.637+0000 7f62ac9ce140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T21:35:48.660 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.657+0000 7f62ac9ce140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T21:35:48.660 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.657+0000 7f4d79bcc140 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T21:35:48.672 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.668+0000 7f221ce85640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b8bc99a9-1c6a-4062-808d-e1fe0423f69b", "id": 0}] 2026-02-22T21:35:48.672 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.668+0000 7f221ce85640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b8bc99a9-1c6a-4062-808d-e1fe0423f69b", "id": 0}] 2026-02-22T21:35:48.673 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.668+0000 7f221ce85640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:48.673 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.668+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b8bc99a9-1c6a-4062-808d-e1fe0423f69b", "id": 0} v 0) -- 0x7f22181b12b0 con 0x7f2218121210 2026-02-22T21:35:48.682 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.677+0000 7f4d79bcc140 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T21:35:48.712 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:48.709+0000 7f62ac9ce140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T21:35:48.733 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:48.729+0000 7f4d79bcc140 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T21:35:48.771 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.767+0000 7f21feffd640 1 -- 10.20.193.140:0/1042482878 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8bc99a9-1c6a-4062-808d-e1fe0423f69b", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f22080429d0 con 0x7f2218121210 2026-02-22T21:35:48.772 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.767+0000 7f21feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b8bc99a9-1c6a-4062-808d-e1fe0423f69b", "id": 0}] 2026-02-22T21:35:48.772 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.767+0000 7f21feffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:48.772 INFO:teuthology.orchestra.run.trial140.stdout:0 2026-02-22T21:35:48.773 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 10 librados: watch_flush enter 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 10 librados: watch_flush exit 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 20 client.4114.objecter shutdown clearing up homeless session... 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 10 client.4114.objecter successfully canceled tick 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 10 monclient: shutdown 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 msgr2=0x7f21e400cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f21e400cc80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2208037680 tx=0x7f220c007e70 comp rx=0 tx=0).stop 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 shutdown_connections 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2218121210 0x7f21e400cc80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f21e4007cd0 0x7f21e400a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 --2- 10.20.193.140:0/1042482878 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f221811d670 0x7f21e4007300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 >> 10.20.193.140:0/1042482878 conn(0x7f2218073410 msgr2=0x7f2218125ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:48.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 shutdown_connections 2026-02-22T21:35:48.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 -- 10.20.193.140:0/1042482878 wait complete. 2026-02-22T21:35:48.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.768+0000 7f221ce85640 1 librados: shutdown 2026-02-22T21:35:48.780 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 2f40aa3d-76d4-49e0-aadd-ac162c75d7d2 1 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: get_monmap_and_config 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: build_initial_monmap 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: monmap: 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:48.880575+0000 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:48.880575+0000 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:48.883 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 Processor -- start 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 -- start start 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: init 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient: _add_conns ranks=[4,2,3,5,0,1] 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fad00073910 0x7fad00073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-b con 0x7fad00073910 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad001161c0 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fad00116d40 0x7fad00118290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-a con 0x7fad00116d40 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad000744a0 con 0x7fad001161c0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fad001283b0 con 0x7fad00116d40 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fad00128530 con 0x7fad00073910 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.879+0000 7fad05e6d640 10 monclient(hunting): authenticate will time out at 1021.506638s 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fad00073910 0x7fad00073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fad00116d40 0x7fad00118290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfeffd640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fad001161c0 0x7fad00116630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.140:34506/0 (socket says 10.20.193.140:34506) 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfeffd640 1 -- 10.20.193.140:0/1554332387 learned_addr learned my addr 10.20.193.140:0/1554332387 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): get_auth_request con 0x7fad00073910 auth_method 0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): get_auth_request con 0x7fad00116d40 auth_method 0 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.884 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.880+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 1 -- 10.20.193.140:0/1554332387 >> v2:10.20.193.51:3300/0 conn(0x7fad00116d40 msgr2=0x7fad00118290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 1 --2- 10.20.193.140:0/1554332387 >> v2:10.20.193.51:3300/0 conn(0x7fad00116d40 0x7fad00118290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 1 -- 10.20.193.140:0/1554332387 >> v1:10.20.193.125:6789/0 conn(0x7fad001161c0 legacy=0x7fad00116630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 1 monclient: found mon.noname-b 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.885 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.881+0000 7facff7fe640 1 -- 10.20.193.140:0/1554332387 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad001286e0 con 0x7fad00073910 2026-02-22T21:35:48.886 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.882+0000 7facff7fe640 1 --2- 10.20.193.140:0/1554332387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad00073cf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fad000768b0 tx=0x7facec02ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d44b1f264eab1ee1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 1 -- 10.20.193.140:0/1554332387 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec03c070 con 0x7fad00073910 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 10 monclient: dump: 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.893 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:48.894 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7facfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7fad05e6d640 5 monclient: authenticate success, global_id 4117 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.883+0000 7fad05e6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 1 -- 10.20.193.140:0/1554332387 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec02fa60 con 0x7fad00073910 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 1 -- 10.20.193.140:0/1554332387 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec02fe30 con 0x7fad00073910 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7facfe7fc640 10 monclient: dump: 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.895 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 4 set_mon_vals no callback set 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 10 monclient: shutdown 2026-02-22T21:35:48.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- 10.20.193.140:0/1554332387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 msgr2=0x7fad00073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1554332387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad00073cf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fad000768b0 tx=0x7facec02ed50 comp rx=0 tx=0).stop 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- 10.20.193.140:0/1554332387 shutdown_connections 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1554332387 >> v2:10.20.193.51:3300/0 conn(0x7fad00116d40 0x7fad00118290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1554332387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad00073cf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- 10.20.193.140:0/1554332387 >> 10.20.193.140:0/1554332387 conn(0x7fad0006f5f0 msgr2=0x7fad00071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- 10.20.193.140:0/1554332387 shutdown_connections 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- 10.20.193.140:0/1554332387 wait complete. 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 10 monclient: build_initial_monmap 2026-02-22T21:35:48.897 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 10 monclient: monmap: 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.898 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 librados: starting msgr at 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 librados: starting objecter 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 Processor -- start 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 -- start start 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 librados: setting wanted keys 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 1 librados: calling monclient init 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.884+0000 7fad05e6d640 10 monclient: init 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-b con 0x7fad00073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:48.899 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad001161c0 0x7fad001b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-a con 0x7fad001161c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad00116d40 0x7fad001afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): picked mon.noname-c con 0x7fad00116d40 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fad00127820 con 0x7fad001161c0 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fad001276a0 con 0x7fad00073910 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fad001279a0 con 0x7fad00116d40 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:48.900 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7fad05e6d640 10 monclient(hunting): authenticate will time out at 1021.512145s 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad00116d40 0x7fad001afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfeffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad001161c0 0x7fad001b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.140:46368/0 (socket says 10.20.193.140:46368) 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 1 -- 10.20.193.140:0/1667989663 learned_addr learned my addr 10.20.193.140:0/1667989663 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): get_auth_request con 0x7fad00073910 auth_method 0 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.901 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfffff640 10 monclient(hunting): get_auth_request con 0x7fad00116d40 auth_method 0 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfeffd640 10 monclient(hunting): get_auth_request con 0x7fad001161c0 auth_method 0 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facfeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.902 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.885+0000 7facff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.886+0000 7facfeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 -- 10.20.193.140:0/1667989663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad00116d40 msgr2=0x7fad001afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad00116d40 0x7fad001afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 -- 10.20.193.140:0/1667989663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad001161c0 msgr2=0x7fad001b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:48.903 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad001161c0 0x7fad001b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 monclient: found mon.noname-b 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.887+0000 7facff7fe640 1 -- 10.20.193.140:0/1667989663 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad001b0550 con 0x7fad00073910 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.888+0000 7facff7fe640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7facec009130 tx=0x7facec0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.888+0000 7facdbfff640 10 client.?.objecter ms_handle_connect 0x7fad00073910 2026-02-22T21:35:48.904 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.888+0000 7facdbfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec03c040 con 0x7fad00073910 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: dump: 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.907 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: _finish_auth 0 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: _check_auth_tickets 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:48.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 5 monclient: authenticate success, global_id 4120 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 10 monclient: _renew_subs 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad001b0780 con 0x7fad00073910 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 10 monclient: _renew_subs 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad001b0ab0 con 0x7fad00073910 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7fad05e6d640 1 librados: init done 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec02fa60 con 0x7fad00073910 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:48.909 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec042650 con 0x7fad00073910 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facfdffb640 4 set_mon_vals no callback set 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.889+0000 7facdbfff640 10 monclient: dump: 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.890+0000 7fad05e6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.910 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.890+0000 7fad05e6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.890+0000 7fad05e6d640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.890+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad00073cf0 con 0x7fad00073910 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7facec042b80 con 0x7fad00073910 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7facec04b7e0 con 0x7fad00073910 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 10 client.4120.objecter ms_dispatch2 0x7fad00000b90 osd_map(3..3 src has 1..3) 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2026-02-22T21:35:48.911 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.891+0000 7facdbfff640 20 client.4120.objecter dump_active .. 0 homeless 2026-02-22T21:35:48.934 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.930+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7facec04b9e0 con 0x7fad00073910 2026-02-22T21:35:48.934 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.930+0000 7facdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:48.934 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.930+0000 7facdbfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:48.967 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.964+0000 7fad05e6d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2f40aa3d-76d4-49e0-aadd-ac162c75d7d2", "id": 1}] 2026-02-22T21:35:48.968 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.964+0000 7fad05e6d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2f40aa3d-76d4-49e0-aadd-ac162c75d7d2", "id": 1}] 2026-02-22T21:35:48.968 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.964+0000 7fad05e6d640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:48.968 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:48.964+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2f40aa3d-76d4-49e0-aadd-ac162c75d7d2", "id": 1} v 0) -- 0x7fad001bcc20 con 0x7fad00073910 2026-02-22T21:35:49.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.011+0000 7facdbfff640 1 -- 10.20.193.140:0/1667989663 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f40aa3d-76d4-49e0-aadd-ac162c75d7d2", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7facec04bc10 con 0x7fad00073910 2026-02-22T21:35:49.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.011+0000 7facdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2f40aa3d-76d4-49e0-aadd-ac162c75d7d2", "id": 1}] 2026-02-22T21:35:49.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.011+0000 7facdbfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:49.015 INFO:teuthology.orchestra.run.trial140.stdout:1 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 10 librados: watch_flush enter 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 10 librados: watch_flush exit 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 20 client.4120.objecter shutdown clearing up homeless session... 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 10 client.4120.objecter successfully canceled tick 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 10 monclient: shutdown 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 msgr2=0x7fad001b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7facec009130 tx=0x7facec0047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 shutdown_connections 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad00116d40 0x7fad001afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad001161c0 0x7fad001b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 --2- 10.20.193.140:0/1667989663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad00073910 0x7fad001b5be0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 >> 10.20.193.140:0/1667989663 conn(0x7fad0006f5f0 msgr2=0x7fad000723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 shutdown_connections 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 -- 10.20.193.140:0/1667989663 wait complete. 2026-02-22T21:35:49.016 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.012+0000 7fad05e6d640 1 librados: shutdown 2026-02-22T21:35:49.024 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new fbb12d84-86d8-45a8-bbd9-23b4d202e27a 2 2026-02-22T21:35:49.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: get_monmap_and_config 2026-02-22T21:35:49.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: monmap: 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:49.085761+0000 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:49.085761+0000 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:49.091 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 Processor -- start 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 -- start start 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: init 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient: _add_conns ranks=[1,3,4,2,5,0] 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f7e94073910 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7e94117280 0x7f7e941176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-a con 0x7f7e94117280 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f7e94117c30 0x7f7e941180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7e94117c30 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e94118740 con 0x7f7e94073910 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.092 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f7e941283b0 con 0x7f7e94117280 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f7e94128530 con 0x7f7e94117c30 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.084+0000 7f7e995a9640 10 monclient(hunting): authenticate will time out at 1021.711840s 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e937fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f7e94117c30 0x7f7e941180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7e94117280 0x7f7e941176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7e94117280 0x7f7e941176f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:35948/0 (socket says 10.20.193.140:35948) 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 1 -- 10.20.193.140:0/2342394669 learned_addr learned my addr 10.20.193.140:0/2342394669 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): get_auth_request con 0x7f7e94117280 auth_method 0 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e937fe640 10 monclient(hunting): get_auth_request con 0x7f7e94117c30 auth_method 0 2026-02-22T21:35:49.093 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e937fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e937fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e92ffd640 1 -- 10.20.193.140:0/2342394669 >> v1:10.20.193.68:6789/0 conn(0x7f7e94073910 legacy=0x7f7e94073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e92ffd640 1 -- 10.20.193.140:0/2342394669 >> v1:10.20.193.68:6789/0 conn(0x7f7e94073910 legacy=0x7f7e94073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e92ffd640 1 --1- 10.20.193.140:0/2342394669 >> v1:10.20.193.68:6789/0 conn(0x7f7e94073910 0x7f7e94073cf0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.085+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.094 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 1 -- 10.20.193.140:0/2342394669 >> v2:10.20.193.68:3300/0 conn(0x7f7e94117c30 msgr2=0x7f7e941180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 1 --2- 10.20.193.140:0/2342394669 >> v2:10.20.193.68:3300/0 conn(0x7f7e94117c30 0x7f7e941180c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 1 -- 10.20.193.140:0/2342394669 >> v1:10.20.193.68:6789/0 conn(0x7f7e94073910 legacy=0x7f7e94073cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.086+0000 7f7e927fc640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.087+0000 7f7e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.087+0000 7f7e927fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.087+0000 7f7e927fc640 1 -- 10.20.193.140:0/2342394669 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e941286e0 con 0x7f7e94117280 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.088+0000 7f7e927fc640 1 --2- 10.20.193.140:0/2342394669 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94117280 0x7f7e941176f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e940745b0 tx=0x7f7e7c02ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e85aa9c18f382d4 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 1 -- 10.20.193.140:0/2342394669 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c03c070 con 0x7f7e94117280 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: dump: 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 5 monclient: authenticate success, global_id 4119 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 1 -- 10.20.193.140:0/2342394669 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7c02fa90 con 0x7f7e94117280 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 1 -- 10.20.193.140:0/2342394669 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c02fe60 con 0x7f7e94117280 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e91ffb640 10 monclient: dump: 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.104 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 4 set_mon_vals no callback set 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 10 monclient: shutdown 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 -- 10.20.193.140:0/2342394669 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94117280 msgr2=0x7f7e941176f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2342394669 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94117280 0x7f7e941176f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e940745b0 tx=0x7f7e7c02ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 -- 10.20.193.140:0/2342394669 shutdown_connections 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2342394669 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94117280 0x7f7e941176f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2342394669 >> v2:10.20.193.68:3300/0 conn(0x7f7e94117c30 0x7f7e941180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 -- 10.20.193.140:0/2342394669 >> 10.20.193.140:0/2342394669 conn(0x7f7e9406f5f0 msgr2=0x7f7e94071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 -- 10.20.193.140:0/2342394669 shutdown_connections 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 -- 10.20.193.140:0/2342394669 wait complete. 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 10 monclient: monmap: 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:49.106 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 librados: starting msgr at 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 librados: starting objecter 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.090+0000 7f7e995a9640 1 Processor -- start 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 -- start start 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 librados: setting wanted keys 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 librados: calling monclient init 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient: init 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 0x7f7e941b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-a con 0x7f7e94073910 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 0x7f7e941b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-b con 0x7f7e94117280 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:49.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7e94117c30 0x7f7e941afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): picked mon.noname-c con 0x7f7e94117c30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f7e94127820 con 0x7f7e94073910 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f7e941276a0 con 0x7f7e94117280 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7e941279a0 con 0x7f7e94117c30 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e995a9640 10 monclient(hunting): authenticate will time out at 1021.718182s 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 0x7f7e941b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e937fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7e94117c30 0x7f7e941afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e927fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 0x7f7e941b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 0x7f7e941b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:35958/0 (socket says 10.20.193.140:35958) 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 1 -- 10.20.193.140:0/2670596446 learned_addr learned my addr 10.20.193.140:0/2670596446 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 10 monclient(hunting): get_auth_request con 0x7f7e94073910 auth_method 0 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e92ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e927fc640 10 monclient(hunting): get_auth_request con 0x7f7e94117280 auth_method 0 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e927fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e927fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e937fe640 10 monclient(hunting): get_auth_request con 0x7f7e94117c30 auth_method 0 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e937fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.091+0000 7f7e937fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.109 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.092+0000 7f7e937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 -- 10.20.193.140:0/2670596446 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7e94117c30 msgr2=0x7f7e941afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7e94117c30 0x7f7e941afcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 -- 10.20.193.140:0/2670596446 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 msgr2=0x7f7e941b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 0x7f7e941b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.110 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 monclient: found mon.noname-b 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.093+0000 7f7e927fc640 1 -- 10.20.193.140:0/2670596446 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e941b0590 con 0x7f7e94117280 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.094+0000 7f7e927fc640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 0x7f7e941b6290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e940738e0 tx=0x7f7e7c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.094+0000 7f7e737fe640 10 client.?.objecter ms_handle_connect 0x7f7e94117280 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.094+0000 7f7e737fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c03c040 con 0x7f7e94117280 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: dump: 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.111 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.112 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 5 monclient: authenticate success, global_id 4126 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 10 monclient: _renew_subs 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e941b07c0 con 0x7f7e94117280 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 10 monclient: _renew_subs 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e941b0af0 con 0x7f7e94117280 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e995a9640 1 librados: init done 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e7c02f990 con 0x7f7e94117280 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c042810 con 0x7f7e94117280 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:49.113 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e917fa640 4 set_mon_vals no callback set 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.095+0000 7f7e737fe640 10 monclient: dump: 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.096+0000 7f7e995a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.096+0000 7f7e995a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.096+0000 7f7e995a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:49.114 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.096+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e941176f0 con 0x7f7e94117280 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f7e7c0429b0 con 0x7f7e94117280 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f7e7c04b400 con 0x7f7e94117280 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 10 client.4126.objecter ms_dispatch2 0x7f7e94000b90 osd_map(4..4 src has 1..4) 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2026-02-22T21:35:49.115 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.097+0000 7f7e737fe640 20 client.4126.objecter dump_active .. 0 homeless 2026-02-22T21:35:49.115 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:49.103+0000 7f62ac9ce140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T21:35:49.122 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:49.118+0000 7f4d79bcc140 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T21:35:49.128 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:49.125+0000 7f62ac9ce140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T21:35:49.141 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.137+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f7e94118740 con 0x7f7e94117280 2026-02-22T21:35:49.141 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.137+0000 7f7e737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.141 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.137+0000 7f7e737fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:49.144 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:49.141+0000 7f4d79bcc140 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T21:35:49.174 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.170+0000 7f7e995a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fbb12d84-86d8-45a8-bbd9-23b4d202e27a", "id": 2}] 2026-02-22T21:35:49.174 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.170+0000 7f7e995a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fbb12d84-86d8-45a8-bbd9-23b4d202e27a", "id": 2}] 2026-02-22T21:35:49.174 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.170+0000 7f7e995a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:49.174 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.170+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fbb12d84-86d8-45a8-bbd9-23b4d202e27a", "id": 2} v 0) -- 0x7f7e941bcc60 con 0x7f7e94117280 2026-02-22T21:35:49.217 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.212+0000 7f7e737fe640 1 -- 10.20.193.140:0/2670596446 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fbb12d84-86d8-45a8-bbd9-23b4d202e27a", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f7e7c055430 con 0x7f7e94117280 2026-02-22T21:35:49.217 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fbb12d84-86d8-45a8-bbd9-23b4d202e27a", "id": 2}] 2026-02-22T21:35:49.217 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e737fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:49.217 INFO:teuthology.orchestra.run.trial140.stdout:2 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 10 librados: watch_flush enter 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 10 librados: watch_flush exit 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 20 client.4126.objecter shutdown clearing up homeless session... 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 10 client.4126.objecter successfully canceled tick 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 10 monclient: shutdown 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.219 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 msgr2=0x7f7e941b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 0x7f7e941b6290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e940738e0 tx=0x7f7e7c0047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 shutdown_connections 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7e94117c30 0x7f7e941afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7e94117280 0x7f7e941b6290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 --2- 10.20.193.140:0/2670596446 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7e94073910 0x7f7e941b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 >> 10.20.193.140:0/2670596446 conn(0x7f7e9406f5f0 msgr2=0x7f7e94071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 shutdown_connections 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 -- 10.20.193.140:0/2670596446 wait complete. 2026-02-22T21:35:49.220 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.213+0000 7f7e995a9640 1 librados: shutdown 2026-02-22T21:35:49.226 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1 3 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.287+0000 7f81ea1bc640 10 monclient: get_monmap_and_config 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.287+0000 7f81ea1bc640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.287+0000 7f81ea1bc640 10 monclient: monmap: 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:49.289315+0000 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:49.289315+0000 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:49.293 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 Processor -- start 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 -- start start 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient: init 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient: _add_conns ranks=[3,1,0,4,5,2] 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f81e4073a70 0x7f81e4073e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-a con 0x7f81e4073a70 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f81e411a460 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81e411df70 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e4121b10 con 0x7f81e411df70 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e4122d10 con 0x7f81e411a460 2026-02-22T21:35:49.294 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f81e4123f10 con 0x7f81e4073a70 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81ea1bc640 10 monclient(hunting): authenticate will time out at 1021.915392s 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81e37fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f81e4073a70 0x7f81e4073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81e2ffd640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f81e411a460 0x7f81e411c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.140:40476/0 (socket says 10.20.193.140:40476) 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.288+0000 7f81e2ffd640 1 -- 10.20.193.140:0/1133871908 learned_addr learned my addr 10.20.193.140:0/1133871908 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): get_auth_request con 0x7f81e4073a70 auth_method 0 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.295 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.289+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 1 -- 10.20.193.140:0/1133871908 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430370475 0 0) 0x7f81e4122d10 con 0x7f81e411a460 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 10 monclient(hunting): _negotiate 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.296 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 10 monclient(hunting): my global_id is 4129 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e27fc640 1 -- 10.20.193.140:0/1133871908 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81c8003620 con 0x7f81e411a460 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.290+0000 7f81e37fe640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 1 -- 10.20.193.140:0/1133871908 >> v1:10.20.193.68:6789/0 conn(0x7f81e411a460 legacy=0x7f81e411c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 1 -- 10.20.193.140:0/1133871908 >> v1:10.20.193.51:6789/0 conn(0x7f81e411df70 legacy=0x7f81e4120360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.291+0000 7f81e37fe640 1 -- 10.20.193.140:0/1133871908 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e4124090 con 0x7f81e4073a70 2026-02-22T21:35:49.297 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.293+0000 7f81e37fe640 1 --2- 10.20.193.140:0/1133871908 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e4073a70 0x7f81e4073e50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f81d00098a0 tx=0x7f81d002edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2679bdb96b476ee3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.301 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 1 -- 10.20.193.140:0/1133871908 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d003c070 con 0x7f81e4073a70 2026-02-22T21:35:49.301 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.301 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.301 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: dump: 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 1 -- 10.20.193.140:0/1133871908 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d002fae0 con 0x7f81e4073a70 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 5 monclient: authenticate success, global_id 4125 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:49.302 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 1 -- 10.20.193.140:0/1133871908 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0037440 con 0x7f81e4073a70 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81e27fc640 10 monclient: dump: 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 4 set_mon_vals no callback set 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 10 monclient: shutdown 2026-02-22T21:35:49.303 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 -- 10.20.193.140:0/1133871908 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e4073a70 msgr2=0x7f81e4073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/1133871908 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e4073a70 0x7f81e4073e50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f81d00098a0 tx=0x7f81d002edd0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 -- 10.20.193.140:0/1133871908 shutdown_connections 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/1133871908 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e4073a70 0x7f81e4073e50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 -- 10.20.193.140:0/1133871908 >> 10.20.193.140:0/1133871908 conn(0x7f81e406f5f0 msgr2=0x7f81e4071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 -- 10.20.193.140:0/1133871908 shutdown_connections 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 -- 10.20.193.140:0/1133871908 wait complete. 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 10 monclient: monmap: 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.304 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 librados: starting msgr at 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.295+0000 7f81ea1bc640 1 librados: starting objecter 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 Processor -- start 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 -- start start 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 librados: setting wanted keys 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 librados: calling monclient init 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient: init 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81e4073a70 0x7f81e41af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-c con 0x7f81e4073a70 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81e411a460 0x7f81e41af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-b con 0x7f81e411a460 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): picked mon.noname-a con 0x7f81e411df70 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f81e41221a0 con 0x7f81e411df70 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4122020 con 0x7f81e411a460 2026-02-22T21:35:49.305 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4122320 con 0x7f81e4073a70 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81ea1bc640 10 monclient(hunting): authenticate will time out at 1021.923499s 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81e3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81e37fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81e4073a70 0x7f81e41af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.296+0000 7f81e2ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81e411a460 0x7f81e41af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:35980/0 (socket says 10.20.193.140:35980) 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): get_auth_request con 0x7f81e4073a70 auth_method 0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 1 -- 10.20.193.140:0/3996523830 learned_addr learned my addr 10.20.193.140:0/3996523830 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): get_auth_request con 0x7f81e411df70 auth_method 0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): get_auth_request con 0x7f81e411a460 auth_method 0 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.306 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.307 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.297+0000 7f81e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.308 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 10 monclient(hunting): handle_auth_done global_id 4131 payload 824 2026-02-22T21:35:49.308 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 -- 10.20.193.140:0/3996523830 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81e4073a70 msgr2=0x7f81e41af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81e4073a70 0x7f81e41af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 -- 10.20.193.140:0/3996523830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81e411a460 msgr2=0x7f81e41af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81e411a460 0x7f81e41af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.298+0000 7f81e3fff640 1 -- 10.20.193.140:0/3996523830 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e4119110 con 0x7f81e411df70 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.299+0000 7f81e3fff640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f81d800cb90 tx=0x7f81d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.299+0000 7f81bffff640 10 client.?.objecter ms_handle_connect 0x7f81e411df70 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.299+0000 7f81bffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800ad50 con 0x7f81e411df70 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 10 monclient: dump: 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.309 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 5 monclient: authenticate success, global_id 4131 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 10 monclient: _renew_subs 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e41193a0 con 0x7f81e411df70 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 10 monclient: _renew_subs 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e41b61f0 con 0x7f81e411df70 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.300+0000 7f81ea1bc640 1 librados: init done 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81ea1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81ea1bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81ea1bc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.310 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e4073ed0 con 0x7f81e411df70 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d8003a70 con 0x7f81e411df70 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d80050d0 con 0x7f81e411df70 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81e1ffb640 4 set_mon_vals no callback set 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.301+0000 7f81bffff640 10 monclient: dump: 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.311 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.359 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f81d8005370 con 0x7f81e411df70 2026-02-22T21:35:49.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f81d8012420 con 0x7f81e411df70 2026-02-22T21:35:49.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 10 client.4131.objecter ms_dispatch2 0x7f81e4000b90 osd_map(5..5 src has 1..5) 2026-02-22T21:35:49.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 3 client.4131.objecter handle_osd_map got epochs [5,5] > 0 2026-02-22T21:35:49.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 3 client.4131.objecter handle_osd_map decoding full epoch 5 2026-02-22T21:35:49.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.355+0000 7f81bffff640 20 client.4131.objecter dump_active .. 0 homeless 2026-02-22T21:35:49.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.358+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f81d8005820 con 0x7f81e411df70 2026-02-22T21:35:49.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.358+0000 7f81bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.358+0000 7f81bffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:49.396 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.392+0000 7f81ea1bc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1", "id": 3}] 2026-02-22T21:35:49.396 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.392+0000 7f81ea1bc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1", "id": 3}] 2026-02-22T21:35:49.396 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.392+0000 7f81ea1bc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.396 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.392+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1", "id": 3} v 0) -- 0x7f81e4070b50 con 0x7f81e411df70 2026-02-22T21:35:49.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.406+0000 7f81bffff640 1 -- 10.20.193.140:0/3996523830 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f81e4070b50 con 0x7f81e411df70 2026-02-22T21:35:49.411 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.406+0000 7f81bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1", "id": 3}] 2026-02-22T21:35:49.411 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.406+0000 7f81bffff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:49.411 INFO:teuthology.orchestra.run.trial140.stdout:3 2026-02-22T21:35:49.411 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 10 librados: watch_flush enter 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 10 librados: watch_flush exit 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 20 client.4131.objecter shutdown clearing up homeless session... 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 10 client.4131.objecter successfully canceled tick 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 10 monclient: shutdown 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 msgr2=0x7f81e4118850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f81d800cb90 tx=0x7f81d800a430 comp rx=0 tx=0).stop 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 shutdown_connections 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81e411df70 0x7f81e4118850 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81e411a460 0x7f81e41af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 --2- 10.20.193.140:0/3996523830 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81e4073a70 0x7f81e41af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 >> 10.20.193.140:0/3996523830 conn(0x7f81e406f5f0 msgr2=0x7f81e4120360 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 shutdown_connections 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 -- 10.20.193.140:0/3996523830 wait complete. 2026-02-22T21:35:49.412 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.407+0000 7f81ea1bc640 1 librados: shutdown 2026-02-22T21:35:49.419 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 6679621e-8d0d-405b-b6db-ef0d22409ae8 4 2026-02-22T21:35:49.471 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:49.468+0000 7f62ac9ce140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T21:35:49.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: get_monmap_and_config 2026-02-22T21:35:49.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: monmap: 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:49.482524+0000 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:49.482524+0000 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:49.487 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 Processor -- start 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 -- start start 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: init 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient: _add_conns ranks=[3,0,5,2,4,1] 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9934073910 0x7f9934073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.488 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-a con 0x7f9934073910 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9934074230 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f99341200b0 0x7f9934122570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-c con 0x7f99341200b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9934122cc0 con 0x7f9934074230 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f9934123ee0 con 0x7f9934073910 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f9934124060 con 0x7f99341200b0 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.481+0000 7f993af2c640 10 monclient(hunting): authenticate will time out at 1022.108620s 2026-02-22T21:35:49.489 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f992bfff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f9934074230 0x7f993411e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.140:34290/0 (socket says 10.20.193.140:34290) 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f992bfff640 1 -- 10.20.193.140:0/2518815876 learned_addr learned my addr 10.20.193.140:0/2518815876 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f9938ca1640 1 --2- 10.20.193.140:0/2518815876 >> v2:10.20.193.51:3300/0 conn(0x7f9934073910 0x7f9934073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f9938ca1640 10 monclient(hunting): get_auth_request con 0x7f9934073910 auth_method 0 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f9938ca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f9938ca1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.490 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.482+0000 7f9938ca1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.491 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.483+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.491 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.483+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.491 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.483+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.491 INFO:tasks.ceph.mgr.x.trial140.stderr:2026-02-22T21:35:49.483+0000 7f4d79bcc140 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T21:35:49.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 10 monclient(hunting): handle_auth_done global_id 4134 payload 274 2026-02-22T21:35:49.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 1 -- 10.20.193.140:0/2518815876 >> v2:10.20.193.125:3300/0 conn(0x7f99341200b0 msgr2=0x7f9934122570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:49.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 1 --2- 10.20.193.140:0/2518815876 >> v2:10.20.193.125:3300/0 conn(0x7f99341200b0 0x7f9934122570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 1 -- 10.20.193.140:0/2518815876 >> v1:10.20.193.51:6789/0 conn(0x7f9934074230 legacy=0x7f993411e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.484+0000 7f9938ca1640 1 -- 10.20.193.140:0/2518815876 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9934124210 con 0x7f9934073910 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.486+0000 7f9938ca1640 1 --2- 10.20.193.140:0/2518815876 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9934073910 0x7f9934073cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9934120080 tx=0x7f992402ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee253eef65fc8625 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 1 -- 10.20.193.140:0/2518815876 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992403c070 con 0x7f9934073910 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: dump: 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.499 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 1 -- 10.20.193.140:0/2518815876 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992402fa60 con 0x7f9934073910 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 5 monclient: authenticate success, global_id 4134 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 1 -- 10.20.193.140:0/2518815876 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992402fe30 con 0x7f9934073910 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 4 set_mon_vals no callback set 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f992b7fe640 10 monclient: dump: 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.501 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 10 monclient: shutdown 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 -- 10.20.193.140:0/2518815876 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9934073910 msgr2=0x7f9934073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 --2- 10.20.193.140:0/2518815876 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9934073910 0x7f9934073cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9934120080 tx=0x7f992402ed50 comp rx=0 tx=0).stop 2026-02-22T21:35:49.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 -- 10.20.193.140:0/2518815876 shutdown_connections 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 --2- 10.20.193.140:0/2518815876 >> v2:10.20.193.125:3300/0 conn(0x7f99341200b0 0x7f9934122570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 --2- 10.20.193.140:0/2518815876 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9934073910 0x7f9934073cf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 -- 10.20.193.140:0/2518815876 >> 10.20.193.140:0/2518815876 conn(0x7f993406f5f0 msgr2=0x7f9934071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 -- 10.20.193.140:0/2518815876 shutdown_connections 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.487+0000 7f993af2c640 1 -- 10.20.193.140:0/2518815876 wait complete. 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient: monmap: 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:49.503 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 librados: starting msgr at 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 librados: starting objecter 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 Processor -- start 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 -- start start 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 librados: setting wanted keys 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 librados: calling monclient init 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient: init 2026-02-22T21:35:49.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9934073910 0x7f99341abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-b con 0x7f9934073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9934074230 0x7f99341ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-c con 0x7f9934074230 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): picked mon.noname-a con 0x7f99341200b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9934123310 con 0x7f99341200b0 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9934123190 con 0x7f9934073910 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9934123490 con 0x7f9934074230 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f993af2c640 10 monclient(hunting): authenticate will time out at 1022.115503s 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f99394a2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.488+0000 7f9938ca1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9934073910 0x7f99341abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:35996/0 (socket says 10.20.193.140:35996) 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 1 -- 10.20.193.140:0/2054004443 learned_addr learned my addr 10.20.193.140:0/2054004443 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): get_auth_request con 0x7f99341200b0 auth_method 0 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): get_auth_request con 0x7f9934073910 auth_method 0 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f99394a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.489+0000 7f9938ca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 -- 10.20.193.140:0/2054004443 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9934074230 msgr2=0x7f99341ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9934074230 0x7f99341ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 -- 10.20.193.140:0/2054004443 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9934073910 msgr2=0x7f99341abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9934073910 0x7f99341abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.490+0000 7f99394a2640 1 -- 10.20.193.140:0/2054004443 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99341b0b00 con 0x7f99341200b0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.491+0000 7f99394a2640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f993000b9e0 tx=0x7f993000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.491+0000 7f99297fa640 10 client.?.objecter ms_handle_connect 0x7f99341200b0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.491+0000 7f99297fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000c6b0 con 0x7f99341200b0 2026-02-22T21:35:49.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: dump: 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.509 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9930010070 con 0x7f99341200b0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 5 monclient: authenticate success, global_id 4140 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _renew_subs 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99341b0df0 con 0x7f99341200b0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000cab0 con 0x7f99341200b0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.510 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f99297fa640 10 monclient: dump: 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f992a7fc640 4 set_mon_vals no callback set 2026-02-22T21:35:49.511 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _renew_subs 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99341b8570 con 0x7f99341200b0 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 1 librados: init done 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.512 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.493+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9934073cf0 con 0x7f99341200b0 2026-02-22T21:35:49.513 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f9930004070 con 0x7f99341200b0 2026-02-22T21:35:49.513 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f993001e900 con 0x7f99341200b0 2026-02-22T21:35:49.513 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 10 client.4140.objecter ms_dispatch2 0x7f9934000b90 osd_map(6..6 src has 1..6) 2026-02-22T21:35:49.513 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2026-02-22T21:35:49.514 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2026-02-22T21:35:49.514 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.495+0000 7f99297fa640 20 client.4140.objecter dump_active .. 0 homeless 2026-02-22T21:35:49.538 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.534+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f9930015470 con 0x7f99341200b0 2026-02-22T21:35:49.538 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.534+0000 7f99297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.538 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.534+0000 7f99297fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:49.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.566+0000 7f993af2c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6679621e-8d0d-405b-b6db-ef0d22409ae8", "id": 4}] 2026-02-22T21:35:49.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.566+0000 7f993af2c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6679621e-8d0d-405b-b6db-ef0d22409ae8", "id": 4}] 2026-02-22T21:35:49.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.566+0000 7f993af2c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.570 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.566+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6679621e-8d0d-405b-b6db-ef0d22409ae8", "id": 4} v 0) -- 0x7f99341b87a0 con 0x7f99341200b0 2026-02-22T21:35:49.664 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.660+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mgrmap(e 2) ==== 49971+0+0 (secure 0 0 0) 0x7f99300250b0 con 0x7f99341200b0 2026-02-22T21:35:49.730 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f99297fa640 1 -- 10.20.193.140:0/2054004443 <== mon.0 v2:10.20.193.51:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6679621e-8d0d-405b-b6db-ef0d22409ae8", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f99300519b0 con 0x7f99341200b0 2026-02-22T21:35:49.731 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f99297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6679621e-8d0d-405b-b6db-ef0d22409ae8", "id": 4}] 2026-02-22T21:35:49.731 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f99297fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:49.731 INFO:teuthology.orchestra.run.trial140.stdout:4 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 10 librados: watch_flush enter 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 10 librados: watch_flush exit 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 20 client.4140.objecter shutdown clearing up homeless session... 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 10 client.4140.objecter successfully canceled tick 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 10 monclient: shutdown 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 msgr2=0x7f99341b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f993000b9e0 tx=0x7f993000beb0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.733 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 shutdown_connections 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99341200b0 0x7f99341b0240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9934074230 0x7f99341ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 --2- 10.20.193.140:0/2054004443 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9934073910 0x7f99341abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 >> 10.20.193.140:0/2054004443 conn(0x7f993406f5f0 msgr2=0x7f9934071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 shutdown_connections 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 -- 10.20.193.140:0/2054004443 wait complete. 2026-02-22T21:35:49.734 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.727+0000 7f993af2c640 1 librados: shutdown 2026-02-22T21:35:49.741 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new c044485c-e401-4a33-885e-8f573934ea0e 5 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: get_monmap_and_config 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: monmap: 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:49.802872+0000 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:49.802872+0000 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:49.805 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 Processor -- start 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 -- start start 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: init 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient: _add_conns ranks=[0,3,1,4,5,2] 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9a8073910 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc9a811d960 0x7fc9a811fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a811d960 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fc9a8120290 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a8123e60 con 0x7fc9a8073910 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a8125080 con 0x7fc9a8120290 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a8126280 con 0x7fc9a811d960 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.801+0000 7fc9ad32c640 10 monclient(hunting): authenticate will time out at 1022.428970s 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.802+0000 7fc9a6ffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fc9a8073910 0x7fc9a8073cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.140:34298/0 (socket says 10.20.193.140:34298) 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.802+0000 7fc9a6ffd640 1 -- 10.20.193.140:0/2709948408 learned_addr learned my addr 10.20.193.140:0/2709948408 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.802+0000 7fc9a67fc640 1 --2- 10.20.193.140:0/2709948408 >> v2:10.20.193.51:3300/0 conn(0x7fc9a811d960 0x7fc9a811fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): get_auth_request con 0x7fc9a811d960 auth_method 0 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.803+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 1 -- 10.20.193.140:0/2709948408 >> v1:10.20.193.68:6789/0 conn(0x7fc9a8120290 legacy=0x7fc9a8122680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 1 -- 10.20.193.140:0/2709948408 >> v1:10.20.193.51:6789/0 conn(0x7fc9a8073910 legacy=0x7fc9a8073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.804+0000 7fc9a67fc640 1 -- 10.20.193.140:0/2709948408 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a8126400 con 0x7fc9a811d960 2026-02-22T21:35:49.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.806+0000 7fc9a67fc640 1 --2- 10.20.193.140:0/2709948408 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a811d960 0x7fc9a811fd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9900098a0 tx=0x7fc99002ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206cedf61d424fab server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 1 -- 10.20.193.140:0/2709948408 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99003c070 con 0x7fc9a811d960 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: dump: 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 5 monclient: authenticate success, global_id 4143 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 1 -- 10.20.193.140:0/2709948408 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99002fa40 con 0x7fc9a811d960 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 1 -- 10.20.193.140:0/2709948408 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99002fe10 con 0x7fc9a811d960 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 4 set_mon_vals no callback set 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9a5ffb640 10 monclient: dump: 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.812 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.813 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.813 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.813 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.813 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.813 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 10 monclient: shutdown 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2709948408 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a811d960 msgr2=0x7fc9a811fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2709948408 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a811d960 0x7fc9a811fd50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9900098a0 tx=0x7fc99002ed80 comp rx=0 tx=0).stop 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2709948408 shutdown_connections 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2709948408 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a811d960 0x7fc9a811fd50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2709948408 >> 10.20.193.140:0/2709948408 conn(0x7fc9a806f5f0 msgr2=0x7fc9a8071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2709948408 shutdown_connections 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2709948408 wait complete. 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 10 monclient: build_initial_monmap 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.808+0000 7fc9ad32c640 10 monclient: monmap: 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 librados: starting msgr at 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 librados: starting objecter 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 Processor -- start 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 -- start start 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 librados: setting wanted keys 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 librados: calling monclient init 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient: init 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9a8073910 0x7fc9a811c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a8073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9a811d960 0x7fc9a811c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a811d960 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a8120290 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8125710 con 0x7fc9a8120290 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8125590 con 0x7fc9a8073910 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a8125890 con 0x7fc9a811d960 2026-02-22T21:35:49.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9ad32c640 10 monclient(hunting): authenticate will time out at 1022.436347s 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a67fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9a811d960 0x7fc9a811c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:36008/0 (socket says 10.20.193.140:36008) 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 1 -- 10.20.193.140:0/2708759238 learned_addr learned my addr 10.20.193.140:0/2708759238 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 10 monclient(hunting): get_auth_request con 0x7fc9a8120290 auth_method 0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.809+0000 7fc9a77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): get_auth_request con 0x7fc9a811d960 auth_method 0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.810+0000 7fc9a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 -- 10.20.193.140:0/2708759238 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9a811d960 msgr2=0x7fc9a811c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9a811d960 0x7fc9a811c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 -- 10.20.193.140:0/2708759238 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9a8073910 msgr2=0x7fc9a811c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9a8073910 0x7fc9a811c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 monclient: found mon.noname-a 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.811+0000 7fc9a77fe640 1 -- 10.20.193.140:0/2708759238 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a81b0bd0 con 0x7fc9a8120290 2026-02-22T21:35:49.816 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.812+0000 7fc9a77fe640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc99800b9e0 tx=0x7fc99800beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:49.816 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.812+0000 7fc9877fe640 10 client.?.objecter ms_handle_connect 0x7fc9a8120290 2026-02-22T21:35:49.816 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.812+0000 7fc9877fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998006e60 con 0x7fc9a8120290 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: dump: 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 5 monclient: authenticate success, global_id 4149 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _renew_subs 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a81b0ec0 con 0x7fc9a8120290 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _renew_subs 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a81b86f0 con 0x7fc9a8120290 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 1 librados: init done 2026-02-22T21:35:49.817 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a8125ca0 con 0x7fc9a8120290 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc998010070 con 0x7fc9a8120290 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99800c910 con 0x7fc9a8120290 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9a57fa640 4 set_mon_vals no callback set 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.813+0000 7fc9877fe640 10 monclient: dump: 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:49.818 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:49.859 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.856+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 2) ==== 49971+0+0 (secure 0 0 0) 0x7fc99801d030 con 0x7fc9a8120290 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fc998014e40 con 0x7fc9a8120290 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 10 client.4149.objecter ms_dispatch2 0x7fc9a8000b90 osd_map(7..7 src has 1..7) 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 20 client.4149.objecter dump_active .. 0 homeless 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7fc998007370 con 0x7fc9a8120290 2026-02-22T21:35:49.863 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:49.864 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.859+0000 7fc9877fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:49.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.891+0000 7fc9ad32c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c044485c-e401-4a33-885e-8f573934ea0e", "id": 5}] 2026-02-22T21:35:49.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.891+0000 7fc9ad32c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c044485c-e401-4a33-885e-8f573934ea0e", "id": 5}] 2026-02-22T21:35:49.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.891+0000 7fc9ad32c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:49.896 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.891+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c044485c-e401-4a33-885e-8f573934ea0e", "id": 5} v 0) -- 0x7fc9a81b88c0 con 0x7fc9a8120290 2026-02-22T21:35:49.976 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9877fe640 1 -- 10.20.193.140:0/2708759238 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c044485c-e401-4a33-885e-8f573934ea0e", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fc998004910 con 0x7fc9a8120290 2026-02-22T21:35:49.976 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c044485c-e401-4a33-885e-8f573934ea0e", "id": 5}] 2026-02-22T21:35:49.976 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9877fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:49.977 INFO:teuthology.orchestra.run.trial140.stdout:5 2026-02-22T21:35:49.978 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 10 librados: watch_flush enter 2026-02-22T21:35:49.978 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 10 librados: watch_flush exit 2026-02-22T21:35:49.978 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 20 client.4149.objecter shutdown clearing up homeless session... 2026-02-22T21:35:49.978 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 10 client.4149.objecter successfully canceled tick 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 10 monclient: shutdown 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 msgr2=0x7fc9a81b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc99800b9e0 tx=0x7fc99800beb0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 shutdown_connections 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9a8120290 0x7fc9a81b0310 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9a811d960 0x7fc9a811c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 --2- 10.20.193.140:0/2708759238 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9a8073910 0x7fc9a811c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 >> 10.20.193.140:0/2708759238 conn(0x7fc9a806f5f0 msgr2=0x7fc9a811dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 shutdown_connections 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 -- 10.20.193.140:0/2708759238 wait complete. 2026-02-22T21:35:49.979 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:49.972+0000 7fc9ad32c640 1 librados: shutdown 2026-02-22T21:35:49.984 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 6908a0e5-ffe4-4cb7-a0ad-43053ea716b1 6 2026-02-22T21:35:50.049 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: get_monmap_and_config 2026-02-22T21:35:50.049 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.049 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: monmap: 2026-02-22T21:35:50.049 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:50.045512+0000 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:50.045512+0000 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:50.050 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 Processor -- start 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 -- start start 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: init 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient: _add_conns ranks=[2,5,0,1,4,3] 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b30073910 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6b30117280 0x7f6b301176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.051 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-c con 0x7f6b30117280 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b30117c30 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b30128370 con 0x7f6b30117c30 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b30129590 con 0x7f6b30073910 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6b3012a790 con 0x7f6b30117280 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.044+0000 7f6b355ed640 10 monclient(hunting): authenticate will time out at 1022.671633s 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.045+0000 7f6b2f7fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f6b30117c30 0x7f6b301180c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.140:34306/0 (socket says 10.20.193.140:34306) 2026-02-22T21:35:50.052 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.045+0000 7f6b2f7fe640 1 -- 10.20.193.140:0/1325037677 learned_addr learned my addr 10.20.193.140:0/1325037677 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:50.054 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 1 -- 10.20.193.140:0/1325037677 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603025600 0 0) 0x7f6b30128370 con 0x7f6b30117c30 2026-02-22T21:35:50.054 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 10 monclient(hunting): _negotiate 2026-02-22T21:35:50.054 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 10 monclient(hunting): my global_id is 4152 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2dffb640 1 -- 10.20.193.140:0/1325037677 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b1c003620 con 0x7f6b30117c30 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.046+0000 7f6b2e7fc640 1 --2- 10.20.193.140:0/1325037677 >> v2:10.20.193.125:3300/0 conn(0x7f6b30117280 0x7f6b301176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f6b30117280 auth_method 0 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.047+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_done global_id 4148 payload 274 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/1325037677 >> v1:10.20.193.125:6789/0 conn(0x7f6b30073910 legacy=0x7f6b30073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/1325037677 >> v1:10.20.193.51:6789/0 conn(0x7f6b30117c30 legacy=0x7f6b301180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 1 monclient: found mon.noname-c 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.048+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/1325037677 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3012a910 con 0x7f6b30117280 2026-02-22T21:35:50.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.050+0000 7f6b2e7fc640 1 --2- 10.20.193.140:0/1325037677 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301176f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b180098a0 tx=0x7f6b1802ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9ab0e3013ebf71 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 1 -- 10.20.193.140:0/1325037677 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1803c070 con 0x7f6b30117280 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: dump: 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.062 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 5 monclient: authenticate success, global_id 4148 2026-02-22T21:35:50.063 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 1 -- 10.20.193.140:0/1325037677 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1802fb70 con 0x7f6b30117280 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 1 -- 10.20.193.140:0/1325037677 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18037440 con 0x7f6b30117280 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 4 set_mon_vals no callback set 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b2dffb640 10 monclient: dump: 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.064 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 10 monclient: shutdown 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 -- 10.20.193.140:0/1325037677 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 msgr2=0x7f6b301176f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 --2- 10.20.193.140:0/1325037677 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301176f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b180098a0 tx=0x7f6b1802ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 -- 10.20.193.140:0/1325037677 shutdown_connections 2026-02-22T21:35:50.065 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 --2- 10.20.193.140:0/1325037677 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301176f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 -- 10.20.193.140:0/1325037677 >> 10.20.193.140:0/1325037677 conn(0x7f6b3006f5f0 msgr2=0x7f6b30071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 -- 10.20.193.140:0/1325037677 shutdown_connections 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.051+0000 7f6b355ed640 1 -- 10.20.193.140:0/1325037677 wait complete. 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient: monmap: 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.066 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 librados: starting msgr at 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 librados: starting objecter 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 Processor -- start 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 -- start start 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 librados: setting wanted keys 2026-02-22T21:35:50.067 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 librados: calling monclient init 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient: init 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6b30073910 0x7f6b301b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-b con 0x7f6b30073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-c con 0x7f6b30117280 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6b30117c30 0x7f6b301279f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): picked mon.noname-a con 0x7f6b30117c30 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30129c20 con 0x7f6b30117c30 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.068 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30129aa0 con 0x7f6b30073910 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6b30129da0 con 0x7f6b30117280 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b355ed640 10 monclient(hunting): authenticate will time out at 1022.679570s 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b2e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.052+0000 7f6b2f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6b30117c30 0x7f6b301279f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.140:56886/0 (socket says 10.20.193.140:56886) 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/679703141 learned_addr learned my addr 10.20.193.140:0/679703141 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:50.069 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): get_auth_request con 0x7f6b30117280 auth_method 0 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b30117c30 auth_method 0 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.070 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.053+0000 7f6b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/679703141 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6b30073910 msgr2=0x7f6b301b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6b30073910 0x7f6b301b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/679703141 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6b30117c30 msgr2=0x7f6b301279f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6b30117c30 0x7f6b301279f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.071 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 monclient: found mon.noname-c 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.054+0000 7f6b2e7fc640 1 -- 10.20.193.140:0/679703141 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b301282b0 con 0x7f6b30117280 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.055+0000 7f6b2e7fc640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009130 tx=0x7f6b180047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.055+0000 7f6b0f7fe640 10 client.?.objecter ms_handle_connect 0x7f6b30117280 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.055+0000 7f6b0f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.056+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1803c040 con 0x7f6b30117280 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.056+0000 7f6b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: dump: 2026-02-22T21:35:50.072 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.073 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b18042630 con 0x7f6b30117280 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18042a20 con 0x7f6b30117280 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b0f7fe640 10 monclient: dump: 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.074 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 5 monclient: authenticate success, global_id 4154 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _renew_subs 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b301284e0 con 0x7f6b30117280 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b2d7fa640 4 set_mon_vals no callback set 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _renew_subs 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.075 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b301bc940 con 0x7f6b30117280 2026-02-22T21:35:50.076 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 1 librados: init done 2026-02-22T21:35:50.076 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.076 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.076 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.076 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.057+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b301176f0 con 0x7f6b30117280 2026-02-22T21:35:50.077 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.063+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 2) ==== 49971+0+0 (secure 0 0 0) 0x7f6b1804b450 con 0x7f6b30117280 2026-02-22T21:35:50.077 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.064+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6b1804b960 con 0x7f6b30117280 2026-02-22T21:35:50.077 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.064+0000 7f6b0f7fe640 10 client.4154.objecter ms_dispatch2 0x7f6b30000b90 osd_map(8..8 src has 1..8) 2026-02-22T21:35:50.077 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.064+0000 7f6b0f7fe640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2026-02-22T21:35:50.078 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.064+0000 7f6b0f7fe640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2026-02-22T21:35:50.078 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.064+0000 7f6b0f7fe640 20 client.4154.objecter dump_active .. 0 homeless 2026-02-22T21:35:50.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.104+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f6b1804acd0 con 0x7f6b30117280 2026-02-22T21:35:50.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.104+0000 7f6b0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.108 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.104+0000 7f6b0f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:50.140 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.136+0000 7f6b355ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6908a0e5-ffe4-4cb7-a0ad-43053ea716b1", "id": 6}] 2026-02-22T21:35:50.140 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.136+0000 7f6b355ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6908a0e5-ffe4-4cb7-a0ad-43053ea716b1", "id": 6}] 2026-02-22T21:35:50.140 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.136+0000 7f6b355ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:50.141 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.136+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6908a0e5-ffe4-4cb7-a0ad-43053ea716b1", "id": 6} v 0) -- 0x7f6b301bcb70 con 0x7f6b30117280 2026-02-22T21:35:50.185 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b0f7fe640 1 -- 10.20.193.140:0/679703141 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6908a0e5-ffe4-4cb7-a0ad-43053ea716b1", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f6b18035340 con 0x7f6b30117280 2026-02-22T21:35:50.185 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6908a0e5-ffe4-4cb7-a0ad-43053ea716b1", "id": 6}] 2026-02-22T21:35:50.185 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b0f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:50.185 INFO:teuthology.orchestra.run.trial140.stdout:6 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 10 librados: watch_flush enter 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 10 librados: watch_flush exit 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 20 client.4154.objecter shutdown clearing up homeless session... 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 10 client.4154.objecter successfully canceled tick 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 10 monclient: shutdown 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 msgr2=0x7f6b301b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009130 tx=0x7f6b180047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 shutdown_connections 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6b30117c30 0x7f6b301279f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6b30117280 0x7f6b301b6250 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 --2- 10.20.193.140:0/679703141 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6b30073910 0x7f6b301b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 >> 10.20.193.140:0/679703141 conn(0x7f6b3006f5f0 msgr2=0x7f6b301144d0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 shutdown_connections 2026-02-22T21:35:50.186 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 -- 10.20.193.140:0/679703141 wait complete. 2026-02-22T21:35:50.187 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.182+0000 7f6b355ed640 1 librados: shutdown 2026-02-22T21:35:50.193 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1 7 2026-02-22T21:35:50.259 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.253+0000 7f656682b640 10 monclient: get_monmap_and_config 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.253+0000 7f656682b640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.253+0000 7f656682b640 10 monclient: monmap: 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:50.255412+0000 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:50.255412+0000 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.260 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 Processor -- start 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 -- start start 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient: init 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient: _add_conns ranks=[2,0,4,3,1,5] 2026-02-22T21:35:50.261 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6560073910 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6560117280 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6560117e00 0x7f6560118290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): picked mon.noname-b con 0x7f6560117e00 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6560128370 con 0x7f6560117280 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6560129590 con 0x7f6560073910 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f656012a790 con 0x7f6560117e00 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f656682b640 10 monclient(hunting): authenticate will time out at 1022.881519s 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f6564da1640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6560117e00 0x7f6560118290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f655ffff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f6560073910 0x7f6560073cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.140:49002/0 (socket says 10.20.193.140:49002) 2026-02-22T21:35:50.262 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.254+0000 7f655ffff640 1 -- 10.20.193.140:0/3860051052 learned_addr learned my addr 10.20.193.140:0/3860051052 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): get_auth_request con 0x7f6560117e00 auth_method 0 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.263 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.255+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 1 -- 10.20.193.140:0/3860051052 >> v1:10.20.193.125:6789/0 conn(0x7f6560073910 legacy=0x7f6560073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 1 -- 10.20.193.140:0/3860051052 >> v1:10.20.193.51:6789/0 conn(0x7f6560117280 legacy=0x7f65601176f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 1 monclient: found mon.noname-b 2026-02-22T21:35:50.270 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.256+0000 7f6564da1640 1 -- 10.20.193.140:0/3860051052 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656012a910 con 0x7f6560117e00 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.257+0000 7f6564da1640 1 --2- 10.20.193.140:0/3860051052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f6560118290 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6554009d90 tx=0x7f655402f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372b220504e12055 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 1 -- 10.20.193.140:0/3860051052 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655403c070 con 0x7f6560117e00 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: dump: 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.271 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f656682b640 5 monclient: authenticate success, global_id 4138 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f656682b640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:50.272 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 1 -- 10.20.193.140:0/3860051052 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6554037440 con 0x7f6560117e00 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 1 -- 10.20.193.140:0/3860051052 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554037810 con 0x7f6560117e00 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f655effd640 10 monclient: dump: 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.273 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f656682b640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.258+0000 7f656682b640 4 set_mon_vals no callback set 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 10 monclient: shutdown 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- 10.20.193.140:0/3860051052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 msgr2=0x7f6560118290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 --2- 10.20.193.140:0/3860051052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f6560118290 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6554009d90 tx=0x7f655402f1e0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- 10.20.193.140:0/3860051052 shutdown_connections 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 --2- 10.20.193.140:0/3860051052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f6560118290 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- 10.20.193.140:0/3860051052 >> 10.20.193.140:0/3860051052 conn(0x7f656006f5f0 msgr2=0x7f6560071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.274 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- 10.20.193.140:0/3860051052 shutdown_connections 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- 10.20.193.140:0/3860051052 wait complete. 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 10 monclient: monmap: 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:50.275 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 librados: starting msgr at 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 librados: starting objecter 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 Processor -- start 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 -- start start 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 librados: setting wanted keys 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 1 librados: calling monclient init 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.259+0000 7f656682b640 10 monclient: init 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6560073910 0x7f65601269f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): picked mon.noname-c con 0x7f6560073910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6560117280 0x7f6560127060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.276 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): picked mon.noname-a con 0x7f6560117280 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): picked mon.noname-b con 0x7f6560117e00 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6560128a00 con 0x7f6560117280 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6560128880 con 0x7f6560117e00 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6560128b80 con 0x7f6560073910 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f656682b640 10 monclient(hunting): authenticate will time out at 1022.887125s 2026-02-22T21:35:50.277 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f655ffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6560073910 0x7f65601269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.140:42822/0 (socket says 10.20.193.140:42822) 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 1 -- 10.20.193.140:0/1125449824 learned_addr learned my addr 10.20.193.140:0/1125449824 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): get_auth_request con 0x7f6560117e00 auth_method 0 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f655ffff640 10 monclient(hunting): get_auth_request con 0x7f6560073910 auth_method 0 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f655ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f655ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f655ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.278 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.260+0000 7f6564da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f655ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f655ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f655ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 -- 10.20.193.140:0/1125449824 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6560073910 msgr2=0x7f65601269f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6560073910 0x7f65601269f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 -- 10.20.193.140:0/1125449824 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6560117280 msgr2=0x7f6560127060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6560117280 0x7f6560127060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 monclient: found mon.noname-b 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.279 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.261+0000 7f6564da1640 1 -- 10.20.193.140:0/1125449824 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65601b60f0 con 0x7f6560117e00 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.262+0000 7f6564da1640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6554009d90 tx=0x7f6554004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.262+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f6560117e00 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.262+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554047020 con 0x7f6560117e00 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: dump: 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.280 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 5 monclient: authenticate success, global_id 4141 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _renew_subs 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65601b6320 con 0x7f6560117e00 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:50.281 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _renew_subs 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65601bc9d0 con 0x7f6560117e00 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 1 librados: init done 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6554037440 con 0x7f6560117e00 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655403c040 con 0x7f6560117e00 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655e7fc640 4 set_mon_vals no callback set 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f655cff9640 10 monclient: dump: 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.282 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.283 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.264+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6560118290 con 0x7f6560117e00 2026-02-22T21:35:50.284 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.270+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 2) ==== 49971+0+0 (secure 0 0 0) 0x7f65540358c0 con 0x7f6560117e00 2026-02-22T21:35:50.284 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.271+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f655404c5c0 con 0x7f6560117e00 2026-02-22T21:35:50.284 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.271+0000 7f655cff9640 10 client.4141.objecter ms_dispatch2 0x7f6560000b90 osd_map(9..9 src has 1..9) 2026-02-22T21:35:50.284 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.271+0000 7f655cff9640 3 client.4141.objecter handle_osd_map got epochs [9,9] > 0 2026-02-22T21:35:50.285 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.271+0000 7f655cff9640 3 client.4141.objecter handle_osd_map decoding full epoch 9 2026-02-22T21:35:50.285 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.271+0000 7f655cff9640 20 client.4141.objecter dump_active .. 0 homeless 2026-02-22T21:35:50.322 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.318+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f6554051030 con 0x7f6560117e00 2026-02-22T21:35:50.322 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.318+0000 7f655cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.323 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.318+0000 7f655cff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:50.354 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.350+0000 7f656682b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1", "id": 7}] 2026-02-22T21:35:50.354 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.350+0000 7f656682b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1", "id": 7}] 2026-02-22T21:35:50.354 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.350+0000 7f656682b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:50.354 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.350+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd new", "uuid": "97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1", "id": 7} v 0) -- 0x7f6560116320 con 0x7f6560117e00 2026-02-22T21:35:50.407 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f655cff9640 1 -- 10.20.193.140:0/1125449824 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f6560116320 con 0x7f6560117e00 2026-02-22T21:35:50.407 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f655cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1", "id": 7}] 2026-02-22T21:35:50.407 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f655cff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:50.408 INFO:teuthology.orchestra.run.trial140.stdout:7 2026-02-22T21:35:50.409 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 10 librados: watch_flush enter 2026-02-22T21:35:50.409 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 10 librados: watch_flush exit 2026-02-22T21:35:50.409 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 20 client.4141.objecter shutdown clearing up homeless session... 2026-02-22T21:35:50.409 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 10 client.4141.objecter successfully canceled tick 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 10 monclient: shutdown 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 msgr2=0x7f65601b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6554009d90 tx=0x7f6554004b40 comp rx=0 tx=0).stop 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 shutdown_connections 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6560117e00 0x7f65601b5830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6560117280 0x7f6560127060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 --2- 10.20.193.140:0/1125449824 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6560073910 0x7f65601269f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 >> 10.20.193.140:0/1125449824 conn(0x7f656006f5f0 msgr2=0x7f6560070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 shutdown_connections 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 -- 10.20.193.140:0/1125449824 wait complete. 2026-02-22T21:35:50.410 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.403+0000 7f656682b640 1 librados: shutdown 2026-02-22T21:35:50.416 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 68963ece-62bb-468d-83aa-461d5d15000d 8 2026-02-22T21:35:50.481 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: get_monmap_and_config 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: monmap: 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:50.475905+0000 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:50.475905+0000 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:50.482 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 1 Processor -- start 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 1 -- start start 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: init 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient: _add_conns ranks=[3,4,5,1,0,2] 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1f1c073910 0x7f1f1c073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.483 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-a con 0x7f1f1c073910 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1f1c074230 0x7f1f1c11e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.474+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-b con 0x7f1f1c074230 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f1f1c11eed0 0x7f1f1c121390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-c con 0x7f1f1c11eed0 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f1f1c121ae0 con 0x7f1f1c073910 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f1f1c121c60 con 0x7f1f1c074230 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f1f1c121e10 con 0x7f1f1c11eed0 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.484 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f23069640 10 monclient(hunting): authenticate will time out at 1023.102023s 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1f1c073910 0x7f1f1c073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1f1c074230 0x7f1f1c11e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f215df640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f1f1c11eed0 0x7f1f1c121390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1f1c073910 0x7f1f1c073cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52220/0 (socket says 10.20.193.140:52220) 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 1 -- 10.20.193.140:0/2099148444 learned_addr learned my addr 10.20.193.140:0/2099148444 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): get_auth_request con 0x7f1f1c073910 auth_method 0 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): get_auth_request con 0x7f1f1c074230 auth_method 0 2026-02-22T21:35:50.485 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f215df640 10 monclient(hunting): get_auth_request con 0x7f1f1c11eed0 auth_method 0 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f215df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f215df640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f215df640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.475+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.486 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 10 monclient(hunting): handle_auth_done global_id 4158 payload 274 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 -- 10.20.193.140:0/2099148444 >> v2:10.20.193.125:3300/0 conn(0x7f1f1c11eed0 msgr2=0x7f1f1c121390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 --2- 10.20.193.140:0/2099148444 >> v2:10.20.193.125:3300/0 conn(0x7f1f1c11eed0 0x7f1f1c121390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 -- 10.20.193.140:0/2099148444 >> v2:10.20.193.68:3300/0 conn(0x7f1f1c074230 msgr2=0x7f1f1c11e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 --2- 10.20.193.140:0/2099148444 >> v2:10.20.193.68:3300/0 conn(0x7f1f1c074230 0x7f1f1c11e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 monclient: found mon.noname-a 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.487 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.476+0000 7f1f20dde640 1 -- 10.20.193.140:0/2099148444 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1c121fc0 con 0x7f1f1c073910 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.479+0000 7f1f20dde640 1 --2- 10.20.193.140:0/2099148444 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c073910 0x7f1f1c073cf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c114310 tx=0x7f1f0c02edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c1639e0de26193 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 1 -- 10.20.193.140:0/2099148444 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c03c070 con 0x7f1f1c073910 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: dump: 2026-02-22T21:35:50.495 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.496 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 1 -- 10.20.193.140:0/2099148444 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0c02fab0 con 0x7f1f1c073910 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 1 -- 10.20.193.140:0/2099148444 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c037440 con 0x7f1f1c073910 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 5 monclient: authenticate success, global_id 4158 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 4 set_mon_vals no callback set 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f137fe640 10 monclient: dump: 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.497 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 10 monclient: shutdown 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- 10.20.193.140:0/2099148444 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c073910 msgr2=0x7f1f1c073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 --2- 10.20.193.140:0/2099148444 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c073910 0x7f1f1c073cf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c114310 tx=0x7f1f0c02edf0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.498 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- 10.20.193.140:0/2099148444 shutdown_connections 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 --2- 10.20.193.140:0/2099148444 >> v2:10.20.193.125:3300/0 conn(0x7f1f1c11eed0 0x7f1f1c121390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 --2- 10.20.193.140:0/2099148444 >> v2:10.20.193.68:3300/0 conn(0x7f1f1c074230 0x7f1f1c11e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 --2- 10.20.193.140:0/2099148444 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c073910 0x7f1f1c073cf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- 10.20.193.140:0/2099148444 >> 10.20.193.140:0/2099148444 conn(0x7f1f1c06f5f0 msgr2=0x7f1f1c071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- 10.20.193.140:0/2099148444 shutdown_connections 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- 10.20.193.140:0/2099148444 wait complete. 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 10 monclient: monmap: 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:50.499 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 librados: starting msgr at 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 librados: starting objecter 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 Processor -- start 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 -- start start 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 librados: setting wanted keys 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 1 librados: calling monclient init 2026-02-22T21:35:50.500 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.481+0000 7f1f23069640 10 monclient: init 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1f1c073910 0x7f1f1c1ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-b con 0x7f1f1c073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1f1c074230 0x7f1f1c1abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-c con 0x7f1f1c074230 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): picked mon.noname-a con 0x7f1f1c11eed0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1226f0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.501 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c122570 con 0x7f1f1c073910 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c122870 con 0x7f1f1c074230 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f23069640 10 monclient(hunting): authenticate will time out at 1023.109114s 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1f1c073910 0x7f1f1c1ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f13fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1f1c074230 0x7f1f1c1abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52226/0 (socket says 10.20.193.140:52226) 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 1 -- 10.20.193.140:0/3978454018 learned_addr learned my addr 10.20.193.140:0/3978454018 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): get_auth_request con 0x7f1f1c11eed0 auth_method 0 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.502 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): get_auth_request con 0x7f1f1c073910 auth_method 0 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f13fff640 10 monclient(hunting): get_auth_request con 0x7f1f1c074230 auth_method 0 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f13fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f13fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f215df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.503 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.482+0000 7f1f20dde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.483+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.483+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.483+0000 7f1f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 10 monclient(hunting): handle_auth_done global_id 4161 payload 824 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 -- 10.20.193.140:0/3978454018 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1f1c074230 msgr2=0x7f1f1c1abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1f1c074230 0x7f1f1c1abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 -- 10.20.193.140:0/3978454018 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1f1c073910 msgr2=0x7f1f1c1ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1f1c073910 0x7f1f1c1ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 monclient: found mon.noname-a 2026-02-22T21:35:50.504 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.484+0000 7f1f215df640 1 -- 10.20.193.140:0/3978454018 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1c1b09b0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f215df640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800efc0 tx=0x7f1f1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 client.?.objecter ms_handle_connect 0x7f1f1c11eed0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18018020 con 0x7f1f1c11eed0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 monclient: dump: 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.505 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 5 monclient: authenticate success, global_id 4161 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 10 monclient: _renew_subs 2026-02-22T21:35:50.506 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f1c1b0ca0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 10 monclient: _renew_subs 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.485+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f1c1b8530 con 0x7f1f1c11eed0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f23069640 1 librados: init done 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f23069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f23069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f23069640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f1c073cf0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f18009b10 con 0x7f1f1c11eed0 2026-02-22T21:35:50.507 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f180146a0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f12ffd640 4 set_mon_vals no callback set 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.486+0000 7f1f117fa640 10 monclient: dump: 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.508 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.509 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.509 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.533 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.529+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 2) ==== 49971+0+0 (secure 0 0 0) 0x7f1f18014840 con 0x7f1f1c11eed0 2026-02-22T21:35:50.534 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.530+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1f1801cca0 con 0x7f1f1c11eed0 2026-02-22T21:35:50.534 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.530+0000 7f1f117fa640 10 client.4161.objecter ms_dispatch2 0x7f1f1c000b90 osd_map(10..10 src has 1..10) 2026-02-22T21:35:50.534 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.530+0000 7f1f117fa640 3 client.4161.objecter handle_osd_map got epochs [10,10] > 0 2026-02-22T21:35:50.534 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.530+0000 7f1f117fa640 3 client.4161.objecter handle_osd_map decoding full epoch 10 2026-02-22T21:35:50.534 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.530+0000 7f1f117fa640 20 client.4161.objecter dump_active .. 0 homeless 2026-02-22T21:35:50.536 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.533+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75931 (secure 0 0 0) 0x7f1f18022070 con 0x7f1f1c11eed0 2026-02-22T21:35:50.536 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.533+0000 7f1f117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.537 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.533+0000 7f1f117fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:50.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.565+0000 7f1f23069640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "68963ece-62bb-468d-83aa-461d5d15000d", "id": 8}] 2026-02-22T21:35:50.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.565+0000 7f1f23069640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "68963ece-62bb-468d-83aa-461d5d15000d", "id": 8}] 2026-02-22T21:35:50.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.565+0000 7f1f23069640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.565+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "68963ece-62bb-468d-83aa-461d5d15000d", "id": 8} v 0) -- 0x7f1f1c1b8760 con 0x7f1f1c11eed0 2026-02-22T21:35:50.608 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f117fa640 1 -- 10.20.193.140:0/3978454018 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "68963ece-62bb-468d-83aa-461d5d15000d", "id": 8}]=0 v11) ==== 112+0+2 (secure 0 0 0) 0x7f1f18010040 con 0x7f1f1c11eed0 2026-02-22T21:35:50.608 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "68963ece-62bb-468d-83aa-461d5d15000d", "id": 8}] 2026-02-22T21:35:50.608 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f117fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:50.609 INFO:teuthology.orchestra.run.trial140.stdout:8 2026-02-22T21:35:50.610 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 10 librados: watch_flush enter 2026-02-22T21:35:50.610 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 10 librados: watch_flush exit 2026-02-22T21:35:50.610 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 20 client.4161.objecter shutdown clearing up homeless session... 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 10 client.4161.objecter successfully canceled tick 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 10 monclient: shutdown 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 msgr2=0x7f1f1c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800efc0 tx=0x7f1f1800c490 comp rx=0 tx=0).stop 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 shutdown_connections 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1f1c11eed0 0x7f1f1c1b00f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1f1c074230 0x7f1f1c1abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 --2- 10.20.193.140:0/3978454018 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1f1c073910 0x7f1f1c1ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 >> 10.20.193.140:0/3978454018 conn(0x7f1f1c06f5f0 msgr2=0x7f1f1c071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.611 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 shutdown_connections 2026-02-22T21:35:50.612 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 -- 10.20.193.140:0/3978454018 wait complete. 2026-02-22T21:35:50.612 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.605+0000 7f1f23069640 1 librados: shutdown 2026-02-22T21:35:50.617 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 6b0f80c5-bcc0-4c72-84d6-af143276a0cf 9 2026-02-22T21:35:50.680 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: get_monmap_and_config 2026-02-22T21:35:50.680 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.680 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: monmap: 2026-02-22T21:35:50.680 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:50.676762+0000 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:50.676762+0000 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:50.681 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 Processor -- start 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 -- start start 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: init 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient: _add_conns ranks=[2,3,0,4,1,5] 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f3c073910 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7f3c117280 0x7f7f3c1176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3c117280 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f3c117c30 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3c128370 con 0x7f7f3c117c30 2026-02-22T21:35:50.682 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3c129590 con 0x7f7f3c073910 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f7f3c12a790 con 0x7f7f3c117280 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.675+0000 7f7f410c4640 10 monclient(hunting): authenticate will time out at 1023.302854s 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.676+0000 7f7f3ad76640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f7f3c073910 0x7f7f3c073cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.140:49018/0 (socket says 10.20.193.140:49018) 2026-02-22T21:35:50.683 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.676+0000 7f7f3ad76640 1 -- 10.20.193.140:0/1264093631 learned_addr learned my addr 10.20.193.140:0/1264093631 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:50.712 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.708+0000 7f7f3a575640 1 --2- 10.20.193.140:0/1264093631 >> v2:10.20.193.51:3300/0 conn(0x7f7f3c117280 0x7f7f3c1176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.715 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.711+0000 7f7f3a575640 10 monclient(hunting): get_auth_request con 0x7f7f3c117280 auth_method 0 2026-02-22T21:35:50.715 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.711+0000 7f7f3a575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.715 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.711+0000 7f7f3a575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.715 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.711+0000 7f7f3a575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.712+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.712+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.716 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.712+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2026-02-22T21:35:50.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 1 -- 10.20.193.140:0/1264093631 >> v1:10.20.193.125:6789/0 conn(0x7f7f3c073910 legacy=0x7f7f3c073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 1 -- 10.20.193.140:0/1264093631 >> v1:10.20.193.51:6789/0 conn(0x7f7f3c117c30 legacy=0x7f7f3c1180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 1 monclient: found mon.noname-a 2026-02-22T21:35:50.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.775+0000 7f7f3a575640 1 -- 10.20.193.140:0/1264093631 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c12a910 con 0x7f7f3c117280 2026-02-22T21:35:50.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.780+0000 7f7f3a575640 1 --2- 10.20.193.140:0/1264093631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1176f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098a0 tx=0x7f7f2402ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b44b18008b13337 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 1 -- 10.20.193.140:0/1264093631 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2403c070 con 0x7f7f3c117280 2026-02-22T21:35:50.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: dump: 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 1 -- 10.20.193.140:0/1264093631 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2402fb70 con 0x7f7f3c117280 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 1 -- 10.20.193.140:0/1264093631 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24037440 con 0x7f7f3c117280 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 5 monclient: authenticate success, global_id 4164 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 4 set_mon_vals no callback set 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f39d74640 10 monclient: dump: 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.802 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: shutdown 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- 10.20.193.140:0/1264093631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 msgr2=0x7f7f3c1176f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 --2- 10.20.193.140:0/1264093631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1176f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098a0 tx=0x7f7f2402ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- 10.20.193.140:0/1264093631 shutdown_connections 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 --2- 10.20.193.140:0/1264093631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1176f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- 10.20.193.140:0/1264093631 >> 10.20.193.140:0/1264093631 conn(0x7f7f3c06f5f0 msgr2=0x7f7f3c071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- 10.20.193.140:0/1264093631 shutdown_connections 2026-02-22T21:35:50.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- 10.20.193.140:0/1264093631 wait complete. 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: build_initial_monmap 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: monmap: 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 librados: starting msgr at 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 librados: starting objecter 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 Processor -- start 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- start start 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 librados: setting wanted keys 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 librados: calling monclient init 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: init 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:35:50.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7f3c073910 0x7f7f3c1b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-b con 0x7f7f3c073910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3c117280 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7f3c117c30 0x7f7f3c1269c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3c117c30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c128a00 con 0x7f7f3c117280 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c128880 con 0x7f7f3c073910 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c128b80 con 0x7f7f3c117c30 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.798+0000 7f7f410c4640 10 monclient(hunting): authenticate will time out at 1023.425820s 2026-02-22T21:35:50.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.802+0000 7f7f3b577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7f3c117c30 0x7f7f3c1269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.803+0000 7f7f3a575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52252/0 (socket says 10.20.193.140:52252) 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 1 -- 10.20.193.140:0/3823361043 learned_addr learned my addr 10.20.193.140:0/3823361043 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 10 monclient(hunting): get_auth_request con 0x7f7f3c117280 auth_method 0 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.807+0000 7f7f3a575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.808+0000 7f7f3ad76640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7f3c073910 0x7f7f3c1b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.808+0000 7f7f3ad76640 10 monclient(hunting): get_auth_request con 0x7f7f3c073910 auth_method 0 2026-02-22T21:35:50.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.808+0000 7f7f3ad76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.808+0000 7f7f3ad76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.808+0000 7f7f3ad76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.813 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.810+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:50.813 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.810+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:50.813 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.810+0000 7f7f3a575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:50.824 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.821+0000 7f7f3b577640 10 monclient(hunting): get_auth_request con 0x7f7f3c117c30 auth_method 0 2026-02-22T21:35:50.824 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.821+0000 7f7f3b577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:50.824 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.821+0000 7f7f3b577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:50.824 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.821+0000 7f7f3b577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 10 monclient(hunting): handle_auth_done global_id 4170 payload 824 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 -- 10.20.193.140:0/3823361043 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7f3c117c30 msgr2=0x7f7f3c1269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7f3c117c30 0x7f7f3c1269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 -- 10.20.193.140:0/3823361043 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7f3c073910 msgr2=0x7f7f3c1b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7f3c073910 0x7f7f3c1b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 10 monclient: _finish_hunting 0 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 monclient: found mon.noname-a 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.829 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.825+0000 7f7f3a575640 1 -- 10.20.193.140:0/3823361043 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c127280 con 0x7f7f3c117280 2026-02-22T21:35:50.831 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.827+0000 7f7f3a575640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009130 tx=0x7f7f240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.831 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.827+0000 7f7f1b7fe640 10 client.?.objecter ms_handle_connect 0x7f7f3c117280 2026-02-22T21:35:50.831 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.827+0000 7f7f1b7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24046020 con 0x7f7f3c117280 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 10 monclient: dump: 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.832 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 5 monclient: authenticate success, global_id 4170 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 10 monclient: _renew_subs 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c127510 con 0x7f7f3c117280 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 10 monclient: _renew_subs 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c1bcb20 con 0x7f7f3c117280 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.828+0000 7f7f410c4640 1 librados: init done 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f410c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f410c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f410c4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:50.833 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f3c1176f0 con 0x7f7f3c117280 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2402fb70 con 0x7f7f3c117280 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2403c040 con 0x7f7f3c117280 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f1b7fe640 10 monclient: dump: 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:50.834 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.829+0000 7f7f39573640 4 set_mon_vals no callback set 2026-02-22T21:35:50.911 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:50.909+0000 7f61c95e5640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-22T21:35:50.932 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:35:50.931+0000 7f61c95e5640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-22T21:35:50.938 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.934+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7f7f24041870 con 0x7f7f3c117280 2026-02-22T21:35:50.938 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.934+0000 7f7f1b7fe640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 0x7f7f0807b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:50.939 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.934+0000 7f7f3ad76640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 0x7f7f0807b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:50.939 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.934+0000 7f7f3ad76640 10 monclient: get_auth_request con 0x7f7f08078c50 auth_method 0 2026-02-22T21:35:50.939 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.935+0000 7f7f3ad76640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 0x7f7f0807b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f30002770 tx=0x7f7f30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:50.939 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.935+0000 7f7f1b7fe640 10 client.4170.objecter ms_handle_connect 0x7f7f08078c50 2026-02-22T21:35:50.945 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 1760+0+0 (secure 0 0 0) 0x7f7f24082f10 con 0x7f7f3c117280 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 10 client.4170.objecter ms_dispatch2 0x7f7f3c000b90 osd_map(11..11 src has 1..11) 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 3 client.4170.objecter handle_osd_map got epochs [11,11] > 0 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 3 client.4170.objecter handle_osd_map decoding full epoch 11 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 20 client.4170.objecter dump_active .. 0 homeless 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7f24050070 con 0x7f7f3c117280 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:50.946 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:50.942+0000 7f7f1b7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:51.033 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.029+0000 7f7f410c4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6b0f80c5-bcc0-4c72-84d6-af143276a0cf", "id": 9}] 2026-02-22T21:35:51.033 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.029+0000 7f7f410c4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6b0f80c5-bcc0-4c72-84d6-af143276a0cf", "id": 9}] 2026-02-22T21:35:51.033 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.029+0000 7f7f410c4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.034 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.029+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6b0f80c5-bcc0-4c72-84d6-af143276a0cf", "id": 9} v 0) -- 0x7f7f3c115310 con 0x7f7f3c117280 2026-02-22T21:35:51.053 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f1b7fe640 1 -- 10.20.193.140:0/3823361043 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b0f80c5-bcc0-4c72-84d6-af143276a0cf", "id": 9}]=0 v12) ==== 112+0+2 (secure 0 0 0) 0x7f7f3c115310 con 0x7f7f3c117280 2026-02-22T21:35:51.053 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6b0f80c5-bcc0-4c72-84d6-af143276a0cf", "id": 9}] 2026-02-22T21:35:51.053 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f1b7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:51.053 INFO:teuthology.orchestra.run.trial140.stdout:9 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 10 librados: watch_flush enter 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 10 librados: watch_flush exit 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 20 client.4170.objecter shutdown clearing up homeless session... 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 10 client.4170.objecter successfully canceled tick 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 msgr2=0x7f7f0807b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 0x7f7f0807b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f30002770 tx=0x7f7f30009040 comp rx=0 tx=0).stop 2026-02-22T21:35:51.055 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 10 monclient: shutdown 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 msgr2=0x7f7f3c1b6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009130 tx=0x7f7f240047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 shutdown_connections 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7f08078c50 0x7f7f0807b110 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7f3c117c30 0x7f7f3c1269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7f3c117280 0x7f7f3c1b6370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 --2- 10.20.193.140:0/3823361043 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7f3c073910 0x7f7f3c1b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 >> 10.20.193.140:0/3823361043 conn(0x7f7f3c06f5f0 msgr2=0x7f7f3c072160 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 shutdown_connections 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 -- 10.20.193.140:0/3823361043 wait complete. 2026-02-22T21:35:51.056 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.049+0000 7f7f410c4640 1 librados: shutdown 2026-02-22T21:35:51.061 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 7c884a35-46d8-45ae-b417-ddb6d976526b 10 2026-02-22T21:35:51.126 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: get_monmap_and_config 2026-02-22T21:35:51.126 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.126 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: monmap: 2026-02-22T21:35:51.126 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.126 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:51.122718+0000 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:51.122718+0000 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 Processor -- start 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 -- start start 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: init 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient: _add_conns ranks=[4,3,0,5,1,2] 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f31c4073910 0x7f31c4073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f31c4073910 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f31c41161c0 0x7f31c4116630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f31c41161c0 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31c4116b70 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:51.127 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c40744a0 con 0x7f31c4116b70 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f31c4117990 con 0x7f31c41161c0 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f31c4117b10 con 0x7f31c4073910 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.121+0000 7f31cb8a9640 10 monclient(hunting): authenticate will time out at 1023.748821s 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f31c4073910 0x7f31c4073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c8e1d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f31c41161c0 0x7f31c4116630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f31c4073910 0x7f31c4073cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.140:42872/0 (socket says 10.20.193.140:42872) 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 1 -- 10.20.193.140:0/228723417 learned_addr learned my addr 10.20.193.140:0/228723417 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): get_auth_request con 0x7f31c4073910 auth_method 0 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c8e1d640 10 monclient(hunting): get_auth_request con 0x7f31c41161c0 auth_method 0 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c8e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c8e1d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.128 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.122+0000 7f31c8e1d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 1 -- 10.20.193.140:0/228723417 >> v2:10.20.193.51:3300/0 conn(0x7f31c41161c0 msgr2=0x7f31c4116630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:51.131 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 1 --2- 10.20.193.140:0/228723417 >> v2:10.20.193.51:3300/0 conn(0x7f31c41161c0 0x7f31c4116630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 1 -- 10.20.193.140:0/228723417 >> v1:10.20.193.51:6789/0 conn(0x7f31c4116b70 legacy=0x7f31c4117280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 1 monclient: found mon.noname-b 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.123+0000 7f31c961e640 1 -- 10.20.193.140:0/228723417 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c4117cc0 con 0x7f31c4073910 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.124+0000 7f31c961e640 1 --2- 10.20.193.140:0/228723417 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4073910 0x7f31c4073cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31c40768b0 tx=0x7f31b402ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb6bea52465a3c0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 1 -- 10.20.193.140:0/228723417 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b403c070 con 0x7f31c4073910 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 10 monclient: dump: 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31cb8a9640 5 monclient: authenticate success, global_id 4153 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.125+0000 7f31cb8a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 1 -- 10.20.193.140:0/228723417 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b402fa90 con 0x7f31c4073910 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 1 -- 10.20.193.140:0/228723417 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b402fe60 con 0x7f31c4073910 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:51.132 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31b3fff640 10 monclient: dump: 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 4 set_mon_vals no callback set 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: shutdown 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- 10.20.193.140:0/228723417 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4073910 msgr2=0x7f31c4073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/228723417 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4073910 0x7f31c4073cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31c40768b0 tx=0x7f31b402ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- 10.20.193.140:0/228723417 shutdown_connections 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/228723417 >> v2:10.20.193.51:3300/0 conn(0x7f31c41161c0 0x7f31c4116630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/228723417 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4073910 0x7f31c4073cf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- 10.20.193.140:0/228723417 >> 10.20.193.140:0/228723417 conn(0x7f31c406f5f0 msgr2=0x7f31c4071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- 10.20.193.140:0/228723417 shutdown_connections 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- 10.20.193.140:0/228723417 wait complete. 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: monmap: 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:51.133 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 librados: starting msgr at 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 librados: starting objecter 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 Processor -- start 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- start start 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 librados: setting wanted keys 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 librados: calling monclient init 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: init 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-a con 0x7f31c4073910 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f31c41161c0 0x7f31c41b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-c con 0x7f31c41161c0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4116b70 0x7f31c41afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): picked mon.noname-b con 0x7f31c4116b70 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f31c41183d0 con 0x7f31c4073910 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4118250 con 0x7f31c4116b70 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f31c4118550 con 0x7f31c41161c0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.126+0000 7f31cb8a9640 10 monclient(hunting): authenticate will time out at 1023.753888s 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4116b70 0x7f31c41afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f31c41161c0 0x7f31c41b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52258/0 (socket says 10.20.193.140:52258) 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 1 -- 10.20.193.140:0/800935720 learned_addr learned my addr 10.20.193.140:0/800935720 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): get_auth_request con 0x7f31c4073910 auth_method 0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): get_auth_request con 0x7f31c4116b70 auth_method 0 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.134 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): get_auth_request con 0x7f31c41161c0 auth_method 0 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c961e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c9e1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.135 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.127+0000 7f31c8e1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 10 monclient(hunting): handle_auth_done global_id 4176 payload 824 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 -- 10.20.193.140:0/800935720 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f31c41161c0 msgr2=0x7f31c41b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f31c41161c0 0x7f31c41b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 -- 10.20.193.140:0/800935720 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4116b70 msgr2=0x7f31c41afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4116b70 0x7f31c41afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 monclient: found mon.noname-a 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.129+0000 7f31c961e640 1 -- 10.20.193.140:0/800935720 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c41b0500 con 0x7f31c4073910 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.130+0000 7f31c961e640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009a70 tx=0x7f31b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.130+0000 7f31b1ffb640 10 client.?.objecter ms_handle_connect 0x7f31c4073910 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.130+0000 7f31b1ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b403c040 con 0x7f31c4073910 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: dump: 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b4042630 con 0x7f31c4073910 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 5 monclient: authenticate success, global_id 4176 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 10 monclient: _renew_subs 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.136 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c41b0730 con 0x7f31c4073910 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4042a20 con 0x7f31c4073910 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b37fe640 4 set_mon_vals no callback set 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31b1ffb640 10 monclient: dump: 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 10 monclient: _renew_subs 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c41bc940 con 0x7f31c4073910 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.131+0000 7f31cb8a9640 1 librados: init done 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.132+0000 7f31cb8a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.132+0000 7f31cb8a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.132+0000 7f31cb8a9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.137 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.132+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c4073cf0 con 0x7f31c4073910 2026-02-22T21:35:51.146 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.142+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7f31b4042bc0 con 0x7f31c4073910 2026-02-22T21:35:51.146 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.143+0000 7f31b1ffb640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 0x7f319407b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.143+0000 7f31c8e1d640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 0x7f319407b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.143+0000 7f31c8e1d640 10 monclient: get_auth_request con 0x7f3194078ca0 auth_method 0 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 10 client.4176.objecter ms_handle_connect 0x7f3194078ca0 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31c8e1d640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 0x7f319407b160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f31c41b5720 tx=0x7f31b8009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 1875+0+0 (secure 0 0 0) 0x7f31b402f990 con 0x7f31c4073910 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 10 client.4176.objecter ms_dispatch2 0x7f31c4000b90 osd_map(12..12 src has 1..12) 2026-02-22T21:35:51.147 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 3 client.4176.objecter handle_osd_map got epochs [12,12] > 0 2026-02-22T21:35:51.148 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 3 client.4176.objecter handle_osd_map decoding full epoch 12 2026-02-22T21:35:51.148 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.144+0000 7f31b1ffb640 20 client.4176.objecter dump_active .. 0 homeless 2026-02-22T21:35:51.248 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.244+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f31c40744a0 con 0x7f31c4073910 2026-02-22T21:35:51.248 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.244+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.249 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.244+0000 7f31b1ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:51.335 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.331+0000 7f31cb8a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7c884a35-46d8-45ae-b417-ddb6d976526b", "id": 10}] 2026-02-22T21:35:51.335 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.331+0000 7f31cb8a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7c884a35-46d8-45ae-b417-ddb6d976526b", "id": 10}] 2026-02-22T21:35:51.336 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.331+0000 7f31cb8a9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.336 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.331+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7c884a35-46d8-45ae-b417-ddb6d976526b", "id": 10} v 0) -- 0x7f31c41bcb70 con 0x7f31c4073910 2026-02-22T21:35:51.372 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.368+0000 7f31b1ffb640 1 -- 10.20.193.140:0/800935720 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c884a35-46d8-45ae-b417-ddb6d976526b", "id": 10}]=0 v13) ==== 113+0+3 (secure 0 0 0) 0x7f31b4087750 con 0x7f31c4073910 2026-02-22T21:35:51.372 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.368+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7c884a35-46d8-45ae-b417-ddb6d976526b", "id": 10}] 2026-02-22T21:35:51.372 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.368+0000 7f31b1ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:51.373 INFO:teuthology.orchestra.run.trial140.stdout:10 2026-02-22T21:35:51.374 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 10 librados: watch_flush enter 2026-02-22T21:35:51.374 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 10 librados: watch_flush exit 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 20 client.4176.objecter shutdown clearing up homeless session... 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 10 client.4176.objecter successfully canceled tick 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 msgr2=0x7f319407b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 0x7f319407b160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f31c41b5720 tx=0x7f31b8009290 comp rx=0 tx=0).stop 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 10 monclient: shutdown 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 msgr2=0x7f31c41b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009a70 tx=0x7f31b40047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 shutdown_connections 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3194078ca0 0x7f319407b160 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.375 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f31c4116b70 0x7f31c41afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f31c41161c0 0x7f31c41b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 --2- 10.20.193.140:0/800935720 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f31c4073910 0x7f31c41b5b90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 >> 10.20.193.140:0/800935720 conn(0x7f31c406f5f0 msgr2=0x7f31c4114610 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 shutdown_connections 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 -- 10.20.193.140:0/800935720 wait complete. 2026-02-22T21:35:51.376 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.369+0000 7f31cb8a9640 1 librados: shutdown 2026-02-22T21:35:51.380 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 9e72c740-4dd0-490b-bad9-483da990d334 11 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 10 monclient: get_monmap_and_config 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 10 monclient: monmap: 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:51.442971+0000 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:51.442971+0000 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:51.445 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 1 Processor -- start 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 1 -- start start 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.441+0000 7fe09d2ea640 10 monclient: init 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient: _add_conns ranks=[3,1,4,2,5,0] 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe098073910 0x7fe098073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-a con 0x7fe098073910 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe0981161c0 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe098116d40 0x7fe098118290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-b con 0x7fe098116d40 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0980744a0 con 0x7fe0981161c0 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fe0981283b0 con 0x7fe098073910 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe098128530 con 0x7fe098116d40 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe09d2ea640 10 monclient(hunting): authenticate will time out at 1024.069075s 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe098073910 0x7fe098073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe0977fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe098116d40 0x7fe098118290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe098073910 0x7fe098073cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52262/0 (socket says 10.20.193.140:52262) 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 1 -- 10.20.193.140:0/2830917529 learned_addr learned my addr 10.20.193.140:0/2830917529 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 10 monclient(hunting): get_auth_request con 0x7fe098073910 auth_method 0 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe0977fe640 10 monclient(hunting): get_auth_request con 0x7fe098116d40 auth_method 0 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe0977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.442+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 1 -- 10.20.193.140:0/2830917529 >> v2:10.20.193.68:3300/0 conn(0x7fe098116d40 msgr2=0x7fe098118290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 1 --2- 10.20.193.140:0/2830917529 >> v2:10.20.193.68:3300/0 conn(0x7fe098116d40 0x7fe098118290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 1 -- 10.20.193.140:0/2830917529 >> v1:10.20.193.68:6789/0 conn(0x7fe0981161c0 legacy=0x7fe098116630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 1 monclient: found mon.noname-a 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.448 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.443+0000 7fe096ffd640 1 -- 10.20.193.140:0/2830917529 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0981286e0 con 0x7fe098073910 2026-02-22T21:35:51.448 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.444+0000 7fe096ffd640 1 --2- 10.20.193.140:0/2830917529 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe098073910 0x7fe098073cf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe0980743e0 tx=0x7fe08c02ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e594f0e39455c9e4 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 1 -- 10.20.193.140:0/2830917529 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c03c070 con 0x7fe098073910 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: dump: 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 1 -- 10.20.193.140:0/2830917529 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe08c02fb30 con 0x7fe098073910 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 5 monclient: authenticate success, global_id 4179 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 1 -- 10.20.193.140:0/2830917529 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c037440 con 0x7fe098073910 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 4 set_mon_vals no callback set 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe095ffb640 10 monclient: dump: 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.450 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.451 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 10 monclient: shutdown 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 -- 10.20.193.140:0/2830917529 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe098073910 msgr2=0x7fe098073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/2830917529 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe098073910 0x7fe098073cf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe0980743e0 tx=0x7fe08c02ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 -- 10.20.193.140:0/2830917529 shutdown_connections 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/2830917529 >> v2:10.20.193.68:3300/0 conn(0x7fe098116d40 0x7fe098118290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/2830917529 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe098073910 0x7fe098073cf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 -- 10.20.193.140:0/2830917529 >> 10.20.193.140:0/2830917529 conn(0x7fe09806f5f0 msgr2=0x7fe098071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 -- 10.20.193.140:0/2830917529 shutdown_connections 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.446+0000 7fe09d2ea640 1 -- 10.20.193.140:0/2830917529 wait complete. 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient: monmap: 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 librados: starting msgr at 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 librados: starting objecter 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 Processor -- start 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 -- start start 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 librados: setting wanted keys 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 librados: calling monclient init 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient: init 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.452 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe098073910 0x7fe0981b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-c con 0x7fe098073910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 0x7fe0981b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-a con 0x7fe0981161c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 0x7fe0981afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): picked mon.noname-b con 0x7fe098116d40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe098127820 con 0x7fe0981161c0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981276a0 con 0x7fe098116d40 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe0981279a0 con 0x7fe098073910 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe09d2ea640 10 monclient(hunting): authenticate will time out at 1024.074524s 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe0967fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 0x7fe0981b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.447+0000 7fe0977fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 0x7fe0981afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe098073910 0x7fe0981b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 0x7fe0981b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52266/0 (socket says 10.20.193.140:52266) 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 1 -- 10.20.193.140:0/158871679 learned_addr learned my addr 10.20.193.140:0/158871679 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): get_auth_request con 0x7fe0981161c0 auth_method 0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): get_auth_request con 0x7fe098116d40 auth_method 0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): get_auth_request con 0x7fe098073910 auth_method 0 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe0977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.453 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.448+0000 7fe096ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 -- 10.20.193.140:0/158871679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe098073910 msgr2=0x7fe0981b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe098073910 0x7fe0981b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 -- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 msgr2=0x7fe0981b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 0x7fe0981b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 monclient: found mon.noname-b 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.449+0000 7fe0977fe640 1 -- 10.20.193.140:0/158871679 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0981b04d0 con 0x7fe098116d40 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.450+0000 7fe0977fe640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 0x7fe0981afc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe08800efc0 tx=0x7fe08800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.450+0000 7fe0777fe640 10 client.?.objecter ms_handle_connect 0x7fe098116d40 2026-02-22T21:35:51.454 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.450+0000 7fe0777fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088018020 con 0x7fe098116d40 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: dump: 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe088014480 con 0x7fe098116d40 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088009b10 con 0x7fe098116d40 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0957fa640 4 set_mon_vals no callback set 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe0777fe640 10 monclient: dump: 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.456 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 5 monclient: authenticate success, global_id 4165 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _renew_subs 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0981b0760 con 0x7fe098116d40 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _renew_subs 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0981bcaf0 con 0x7fe098116d40 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 1 librados: init done 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.452+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe098073cf0 con 0x7fe098116d40 2026-02-22T21:35:51.561 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.557+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7fe088014870 con 0x7fe098116d40 2026-02-22T21:35:51.562 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.558+0000 7fe0777fe640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 0x7fe06407b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.562 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.558+0000 7fe096ffd640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 0x7fe06407b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.562 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.558+0000 7fe096ffd640 10 monclient: get_auth_request con 0x7fe064078c50 auth_method 0 2026-02-22T21:35:51.563 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.559+0000 7fe096ffd640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 0x7fe06407b110 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0980742f0 tx=0x7fe08c039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.563 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.559+0000 7fe0777fe640 10 client.4165.objecter ms_handle_connect 0x7fe064078c50 2026-02-22T21:35:51.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.565+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 1990+0+0 (secure 0 0 0) 0x7fe08805edc0 con 0x7fe098116d40 2026-02-22T21:35:51.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.565+0000 7fe0777fe640 10 client.4165.objecter ms_dispatch2 0x7fe098000b90 osd_map(13..13 src has 1..13) 2026-02-22T21:35:51.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.565+0000 7fe0777fe640 3 client.4165.objecter handle_osd_map got epochs [13,13] > 0 2026-02-22T21:35:51.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.565+0000 7fe0777fe640 3 client.4165.objecter handle_osd_map decoding full epoch 13 2026-02-22T21:35:51.569 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.565+0000 7fe0777fe640 20 client.4165.objecter dump_active .. 0 homeless 2026-02-22T21:35:51.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.572+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe088022030 con 0x7fe098116d40 2026-02-22T21:35:51.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.572+0000 7fe0777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.575 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.572+0000 7fe0777fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:51.663 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.658+0000 7fe09d2ea640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9e72c740-4dd0-490b-bad9-483da990d334", "id": 11}] 2026-02-22T21:35:51.663 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.658+0000 7fe09d2ea640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9e72c740-4dd0-490b-bad9-483da990d334", "id": 11}] 2026-02-22T21:35:51.663 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.658+0000 7fe09d2ea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:51.663 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.658+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9e72c740-4dd0-490b-bad9-483da990d334", "id": 11} v 0) -- 0x7fe0981bccc0 con 0x7fe098116d40 2026-02-22T21:35:51.710 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.706+0000 7fe0777fe640 1 -- 10.20.193.140:0/158871679 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e72c740-4dd0-490b-bad9-483da990d334", "id": 11}]=0 v14) ==== 113+0+3 (secure 0 0 0) 0x7fe088010040 con 0x7fe098116d40 2026-02-22T21:35:51.710 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.706+0000 7fe0777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9e72c740-4dd0-490b-bad9-483da990d334", "id": 11}] 2026-02-22T21:35:51.710 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.706+0000 7fe0777fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:51.710 INFO:teuthology.orchestra.run.trial140.stdout:11 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.706+0000 7fe09d2ea640 10 librados: watch_flush enter 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 10 librados: watch_flush exit 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 20 client.4165.objecter shutdown clearing up homeless session... 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 10 client.4165.objecter successfully canceled tick 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 msgr2=0x7fe06407b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 0x7fe06407b110 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0980742f0 tx=0x7fe08c039040 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 10 monclient: shutdown 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 msgr2=0x7fe0981afc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 0x7fe0981afc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe08800efc0 tx=0x7fe08800c490 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 shutdown_connections 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe064078c50 0x7fe06407b110 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe098116d40 0x7fe0981afc10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe0981161c0 0x7fe0981b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 --2- 10.20.193.140:0/158871679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe098073910 0x7fe0981b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 >> 10.20.193.140:0/158871679 conn(0x7fe09806f5f0 msgr2=0x7fe098072480 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 shutdown_connections 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 -- 10.20.193.140:0/158871679 wait complete. 2026-02-22T21:35:51.711 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.707+0000 7fe09d2ea640 1 librados: shutdown 2026-02-22T21:35:51.720 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new e85b2358-a072-4155-a38f-68f53529ebe9 12 2026-02-22T21:35:51.786 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.778+0000 7f660bf44640 10 monclient: get_monmap_and_config 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.778+0000 7f660bf44640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.778+0000 7f660bf44640 10 monclient: monmap: 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:51.780338+0000 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:51.780338+0000 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:51.787 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 Processor -- start 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 -- start start 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient: init 2026-02-22T21:35:51.788 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient: _add_conns ranks=[5,4,0,2,1,3] 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f66040737c0 0x7f6604073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-c con 0x7f66040737c0 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f660411a460 0x7f660411c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-b con 0x7f660411a460 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f660411cd90 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6604120930 con 0x7f660411cd90 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f6604121b30 con 0x7f660411a460 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.789 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6604121cb0 con 0x7f66040737c0 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.779+0000 7f660bf44640 10 monclient(hunting): authenticate will time out at 1024.406625s 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f66040737c0 0x7f6604073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f660411a460 0x7f660411c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f660a4ba640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f660411cd90 0x7f660411f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.140:48298/0 (socket says 10.20.193.140:48298) 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f660a4ba640 1 -- 10.20.193.140:0/3627106541 learned_addr learned my addr 10.20.193.140:0/3627106541 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): get_auth_request con 0x7f66040737c0 auth_method 0 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.790 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): get_auth_request con 0x7f660411a460 auth_method 0 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.780+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 1 -- 10.20.193.140:0/3627106541 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187006018 0 0) 0x7f6604120930 con 0x7f660411cd90 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 10 monclient(hunting): _negotiate 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.791 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 10 monclient(hunting): my global_id is 4185 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6608cb7640 1 -- 10.20.193.140:0/3627106541 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65ec003620 con 0x7f660411cd90 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 1 -- 10.20.193.140:0/3627106541 >> v2:10.20.193.68:3300/0 conn(0x7f660411a460 msgr2=0x7f660411c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 1 --2- 10.20.193.140:0/3627106541 >> v2:10.20.193.68:3300/0 conn(0x7f660411a460 0x7f660411c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 1 -- 10.20.193.140:0/3627106541 >> v1:10.20.193.51:6789/0 conn(0x7f660411cd90 legacy=0x7f660411f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 1 monclient: found mon.noname-c 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.781+0000 7f6609cb9640 1 -- 10.20.193.140:0/3627106541 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6604121e60 con 0x7f66040737c0 2026-02-22T21:35:51.792 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.782+0000 7f6609cb9640 1 --2- 10.20.193.140:0/3627106541 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66040737c0 0x7f6604073ba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6604075140 tx=0x7f65f402ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1994f74c9fc9a2f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 1 -- 10.20.193.140:0/3627106541 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f403c070 con 0x7f66040737c0 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: dump: 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.799 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 1 -- 10.20.193.140:0/3627106541 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f402fa60 con 0x7f66040737c0 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 5 monclient: authenticate success, global_id 4175 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.800 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 1 -- 10.20.193.140:0/3627106541 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f402fe30 con 0x7f66040737c0 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 4 set_mon_vals no callback set 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f6608cb7640 10 monclient: dump: 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.801 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: shutdown 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- 10.20.193.140:0/3627106541 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66040737c0 msgr2=0x7f6604073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- 10.20.193.140:0/3627106541 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66040737c0 0x7f6604073ba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6604075140 tx=0x7f65f402ed50 comp rx=0 tx=0).stop 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- 10.20.193.140:0/3627106541 shutdown_connections 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- 10.20.193.140:0/3627106541 >> v2:10.20.193.68:3300/0 conn(0x7f660411a460 0x7f660411c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- 10.20.193.140:0/3627106541 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66040737c0 0x7f6604073ba0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- 10.20.193.140:0/3627106541 >> 10.20.193.140:0/3627106541 conn(0x7f660406f5f0 msgr2=0x7f6604071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:51.802 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- 10.20.193.140:0/3627106541 shutdown_connections 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- 10.20.193.140:0/3627106541 wait complete. 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: build_initial_monmap 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: monmap: 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:51.803 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 librados: starting msgr at 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 librados: starting objecter 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 Processor -- start 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- start start 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 librados: setting wanted keys 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 librados: calling monclient init 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: init 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66040737c0 0x7f6604112180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.804 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-b con 0x7f66040737c0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 0x7f66041126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-a con 0x7f660411a460 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 0x7f6604112c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient(hunting): picked mon.noname-c con 0x7f660411cd90 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.784+0000 7f660bf44640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6604120fc0 con 0x7f660411a460 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6604120e40 con 0x7f66040737c0 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6604121140 con 0x7f660411cd90 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:51.805 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660bf44640 10 monclient(hunting): authenticate will time out at 1024.412005s 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 0x7f66041126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 0x7f6604112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66040737c0 0x7f6604112180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 0x7f6604112c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.140:57802/0 (socket says 10.20.193.140:57802) 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 1 -- 10.20.193.140:0/2976500380 learned_addr learned my addr 10.20.193.140:0/2976500380 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): get_auth_request con 0x7f660411cd90 auth_method 0 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): get_auth_request con 0x7f660411a460 auth_method 0 2026-02-22T21:35:51.806 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): get_auth_request con 0x7f66040737c0 auth_method 0 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f660a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.807 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f66094b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.785+0000 7f6609cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 -- 10.20.193.140:0/2976500380 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 msgr2=0x7f6604112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 0x7f6604112c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 -- 10.20.193.140:0/2976500380 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66040737c0 msgr2=0x7f6604112180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66040737c0 0x7f6604112180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 10 monclient: _finish_hunting 0 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 monclient: found mon.noname-a 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.808 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.787+0000 7f66094b8640 1 -- 10.20.193.140:0/2976500380 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66041b7960 con 0x7f660411a460 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.788+0000 7f66094b8640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 0x7f66041126c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6604069bc0 tx=0x7f65f800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.788+0000 7f65f27fc640 10 client.?.objecter ms_handle_connect 0x7f660411a460 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.788+0000 7f65f27fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8017070 con 0x7f660411a460 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: dump: 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.809 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f800c9a0 con 0x7f660411a460 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f800cd70 con 0x7f660411a460 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:51.810 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f3fff640 4 set_mon_vals no callback set 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f65f27fc640 10 monclient: dump: 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:51.811 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 5 monclient: authenticate success, global_id 4188 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 10 monclient: _renew_subs 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66041b7bf0 con 0x7f660411a460 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 10 monclient: _renew_subs 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66041a9100 con 0x7f660411a460 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.789+0000 7f660bf44640 1 librados: init done 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.790+0000 7f660bf44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.790+0000 7f660bf44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.812 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.790+0000 7f660bf44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.813 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.790+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6604073ba0 con 0x7f660411a460 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.801+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7f65f8024060 con 0x7f660411a460 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.801+0000 7f65f27fc640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 0x7f65d407b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2105+0+0 (secure 0 0 0) 0x7f65f805e090 con 0x7f660411a460 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 10 client.4188.objecter ms_dispatch2 0x7f6604000b90 osd_map(14..14 src has 1..14) 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 3 client.4188.objecter handle_osd_map got epochs [14,14] > 0 2026-02-22T21:35:51.814 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 3 client.4188.objecter handle_osd_map decoding full epoch 14 2026-02-22T21:35:51.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f6609cb9640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 0x7f65d407b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:51.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f6609cb9640 10 monclient: get_auth_request con 0x7f65d4078c50 auth_method 0 2026-02-22T21:35:51.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 20 client.4188.objecter dump_active .. 0 homeless 2026-02-22T21:35:51.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f6609cb9640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 0x7f65d407b110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65f4039000 tx=0x7f65f4033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:51.815 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.802+0000 7f65f27fc640 10 client.4188.objecter ms_handle_connect 0x7f65d4078c50 2026-02-22T21:35:51.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.904+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f65f8062050 con 0x7f660411a460 2026-02-22T21:35:51.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.904+0000 7f65f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:51.908 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.904+0000 7f65f27fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:51.994 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.990+0000 7f660bf44640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e85b2358-a072-4155-a38f-68f53529ebe9", "id": 12}] 2026-02-22T21:35:51.994 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.990+0000 7f660bf44640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e85b2358-a072-4155-a38f-68f53529ebe9", "id": 12}] 2026-02-22T21:35:51.994 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.990+0000 7f660bf44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:51.994 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:51.990+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e85b2358-a072-4155-a38f-68f53529ebe9", "id": 12} v 0) -- 0x7f66041a93c0 con 0x7f660411a460 2026-02-22T21:35:52.014 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.010+0000 7f65f27fc640 1 -- 10.20.193.140:0/2976500380 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e85b2358-a072-4155-a38f-68f53529ebe9", "id": 12}]=0 v15) ==== 113+0+3 (secure 0 0 0) 0x7f65f8066350 con 0x7f660411a460 2026-02-22T21:35:52.014 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.010+0000 7f65f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e85b2358-a072-4155-a38f-68f53529ebe9", "id": 12}] 2026-02-22T21:35:52.014 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.010+0000 7f65f27fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:52.014 INFO:teuthology.orchestra.run.trial140.stdout:12 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 10 librados: watch_flush enter 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 10 librados: watch_flush exit 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 20 client.4188.objecter shutdown clearing up homeless session... 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 10 client.4188.objecter successfully canceled tick 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 msgr2=0x7f65d407b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 0x7f65d407b110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65f4039000 tx=0x7f65f4033000 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 10 monclient: shutdown 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 msgr2=0x7f66041126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 0x7f66041126c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6604069bc0 tx=0x7f65f800ee20 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 shutdown_connections 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65d4078c50 0x7f65d407b110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f660411cd90 0x7f6604112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f660411a460 0x7f66041126c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 --2- 10.20.193.140:0/2976500380 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66040737c0 0x7f6604112180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 >> 10.20.193.140:0/2976500380 conn(0x7f660406f5f0 msgr2=0x7f66040719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 shutdown_connections 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 -- 10.20.193.140:0/2976500380 wait complete. 2026-02-22T21:35:52.015 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.011+0000 7f660bf44640 1 librados: shutdown 2026-02-22T21:35:52.021 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new ca68e92b-99ca-4584-825d-9c0c601134c9 13 2026-02-22T21:35:52.086 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: get_monmap_and_config 2026-02-22T21:35:52.086 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.086 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: monmap: 2026-02-22T21:35:52.086 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.086 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:52.081788+0000 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:52.081788+0000 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:52.087 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 Processor -- start 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 -- start start 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: init 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient: _add_conns ranks=[0,4,2,3,5,1] 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa79c073910 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa79c11d960 0x7fa79c11fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c11d960 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa79c120290 addr v1:10.20.193.125:6789/0 2026-02-22T21:35:52.088 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c123e60 con 0x7fa79c073910 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c125080 con 0x7fa79c120290 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fa79c126280 con 0x7fa79c11d960 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.080+0000 7fa7a4496640 10 monclient(hunting): authenticate will time out at 1024.707860s 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa79c11d960 0x7fa79c11fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a2a0c640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fa79c120290 0x7fa79c122680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.140:49020/0 (socket says 10.20.193.140:49020) 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a2a0c640 1 -- 10.20.193.140:0/517195240 learned_addr learned my addr 10.20.193.140:0/517195240 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): get_auth_request con 0x7fa79c11d960 auth_method 0 2026-02-22T21:35:52.089 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.090 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.081+0000 7fa7a1a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 10 monclient(hunting): handle_auth_done global_id 4174 payload 274 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 1 -- 10.20.193.140:0/517195240 >> v1:10.20.193.125:6789/0 conn(0x7fa79c120290 legacy=0x7fa79c122680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 1 -- 10.20.193.140:0/517195240 >> v1:10.20.193.51:6789/0 conn(0x7fa79c073910 legacy=0x7fa79c073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 1 monclient: found mon.noname-b 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.097 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.082+0000 7fa7a1a0a640 1 -- 10.20.193.140:0/517195240 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c126400 con 0x7fa79c11d960 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.083+0000 7fa7a1a0a640 1 --2- 10.20.193.140:0/517195240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c11fd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa7900098a0 tx=0x7fa79002ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49324bc3655042b1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.084+0000 7fa7a1209640 1 -- 10.20.193.140:0/517195240 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79003c070 con 0x7fa79c11d960 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.084+0000 7fa7a1209640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: dump: 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.098 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 5 monclient: authenticate success, global_id 4174 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 1 -- 10.20.193.140:0/517195240 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79002fa40 con 0x7fa79c11d960 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 1 -- 10.20.193.140:0/517195240 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79002fe10 con 0x7fa79c11d960 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.099 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a1209640 10 monclient: dump: 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.100 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 4 set_mon_vals no callback set 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: shutdown 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- 10.20.193.140:0/517195240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 msgr2=0x7fa79c11fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 --2- 10.20.193.140:0/517195240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c11fd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa7900098a0 tx=0x7fa79002ed80 comp rx=0 tx=0).stop 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- 10.20.193.140:0/517195240 shutdown_connections 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 --2- 10.20.193.140:0/517195240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c11fd50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- 10.20.193.140:0/517195240 >> 10.20.193.140:0/517195240 conn(0x7fa79c06f5f0 msgr2=0x7fa79c071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- 10.20.193.140:0/517195240 shutdown_connections 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- 10.20.193.140:0/517195240 wait complete. 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.101 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: monmap: 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 librados: starting msgr at 2026-02-22T21:35:52.102 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 librados: starting objecter 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 Processor -- start 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- start start 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 librados: setting wanted keys 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 librados: calling monclient init 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: init 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa79c073910 0x7fa79c1b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-c con 0x7fa79c073910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c1b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c11d960 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.103 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): picked mon.noname-a con 0x7fa79c120290 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1244f0 con 0x7fa79c120290 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c124370 con 0x7fa79c11d960 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c124670 con 0x7fa79c073910 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.085+0000 7fa7a4496640 10 monclient(hunting): authenticate will time out at 1024.712869s 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa79c073910 0x7fa79c1b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52286/0 (socket says 10.20.193.140:52286) 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 1 -- 10.20.193.140:0/2533839420 learned_addr learned my addr 10.20.193.140:0/2533839420 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): get_auth_request con 0x7fa79c120290 auth_method 0 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): get_auth_request con 0x7fa79c073910 auth_method 0 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a2a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.104 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.086+0000 7fa7a220b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 -- 10.20.193.140:0/2533839420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa79c073910 msgr2=0x7fa79c1b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa79c073910 0x7fa79c1b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 -- 10.20.193.140:0/2533839420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 msgr2=0x7fa79c1b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c1b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 monclient: found mon.noname-a 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.088+0000 7fa7a2a0c640 1 -- 10.20.193.140:0/2533839420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c11c740 con 0x7fa79c120290 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.089+0000 7fa7a2a0c640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa79800caa0 tx=0x7fa79800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.089+0000 7fa782ffd640 10 client.?.objecter ms_handle_connect 0x7fa79c120290 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.089+0000 7fa782ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798017070 con 0x7fa79c120290 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: dump: 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 5 monclient: authenticate success, global_id 4191 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 10 monclient: _renew_subs 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c11ca30 con 0x7fa79c120290 2026-02-22T21:35:52.105 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 10 monclient: _renew_subs 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c1b86a0 con 0x7fa79c120290 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a4496640 1 librados: init done 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79800ac40 con 0x7fa79c120290 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79801c640 con 0x7fa79c120290 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa782ffd640 10 monclient: dump: 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.090+0000 7fa7a0a08640 4 set_mon_vals no callback set 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.091+0000 7fa7a4496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.091+0000 7fa7a4496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.091+0000 7fa7a4496640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.106 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.091+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa79c124a80 con 0x7fa79c120290 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.101+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7fa798012400 con 0x7fa79c120290 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 0x7fa76c07b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa7a220b640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 0x7fa76c07b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa7a220b640 10 monclient: get_auth_request con 0x7fa76c078c50 auth_method 0 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2220+0+0 (secure 0 0 0) 0x7fa79805df50 con 0x7fa79c120290 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 10 client.4191.objecter ms_dispatch2 0x7fa79c000b90 osd_map(15..15 src has 1..15) 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 3 client.4191.objecter handle_osd_map got epochs [15,15] > 0 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 3 client.4191.objecter handle_osd_map decoding full epoch 15 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.102+0000 7fa782ffd640 20 client.4191.objecter dump_active .. 0 homeless 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.103+0000 7fa782ffd640 10 client.4191.objecter ms_handle_connect 0x7fa76c078c50 2026-02-22T21:35:52.107 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.103+0000 7fa7a220b640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 0x7fa76c07b110 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa79c11cf40 tx=0x7fa78c0091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.206 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.202+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa798021070 con 0x7fa79c120290 2026-02-22T21:35:52.206 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.202+0000 7fa782ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.206 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.202+0000 7fa782ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:52.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.289+0000 7fa7a4496640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca68e92b-99ca-4584-825d-9c0c601134c9", "id": 13}] 2026-02-22T21:35:52.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.289+0000 7fa7a4496640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca68e92b-99ca-4584-825d-9c0c601134c9", "id": 13}] 2026-02-22T21:35:52.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.289+0000 7fa7a4496640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.293 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.289+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca68e92b-99ca-4584-825d-9c0c601134c9", "id": 13} v 0) -- 0x7fa79c1b8930 con 0x7fa79c120290 2026-02-22T21:35:52.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa782ffd640 1 -- 10.20.193.140:0/2533839420 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca68e92b-99ca-4584-825d-9c0c601134c9", "id": 13}]=0 v16) ==== 113+0+3 (secure 0 0 0) 0x7fa798061f10 con 0x7fa79c120290 2026-02-22T21:35:52.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa782ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca68e92b-99ca-4584-825d-9c0c601134c9", "id": 13}] 2026-02-22T21:35:52.360 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa782ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:52.360 INFO:teuthology.orchestra.run.trial140.stdout:13 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 10 librados: watch_flush enter 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 10 librados: watch_flush exit 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 20 client.4191.objecter shutdown clearing up homeless session... 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 10 client.4191.objecter successfully canceled tick 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 msgr2=0x7fa76c07b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 0x7fa76c07b110 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa79c11cf40 tx=0x7fa78c0091c0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.362 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 10 monclient: shutdown 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 msgr2=0x7fa79c11be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa79800caa0 tx=0x7fa79800cf70 comp rx=0 tx=0).stop 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 shutdown_connections 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa76c078c50 0x7fa76c07b110 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa79c120290 0x7fa79c11be80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa79c11d960 0x7fa79c1b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 --2- 10.20.193.140:0/2533839420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa79c073910 0x7fa79c1b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 >> 10.20.193.140:0/2533839420 conn(0x7fa79c06f5f0 msgr2=0x7fa79c11dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 shutdown_connections 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.356+0000 7fa7a4496640 1 -- 10.20.193.140:0/2533839420 wait complete. 2026-02-22T21:35:52.363 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.357+0000 7fa7a4496640 1 librados: shutdown 2026-02-22T21:35:52.370 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new ce0888c2-dcdf-44c5-a8d4-06a1cd783264 14 2026-02-22T21:35:52.434 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 10 monclient: get_monmap_and_config 2026-02-22T21:35:52.434 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.434 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 10 monclient: monmap: 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:52.432019+0000 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:52.432019+0000 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:52.435 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 1 Processor -- start 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 1 -- start start 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.430+0000 7fd29bf98640 10 monclient: init 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient: _add_conns ranks=[4,5,1,3,2,0] 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd294073910 0x7fd294073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-b con 0x7fd294073910 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd2941161c0 0x7fd294116630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-c con 0x7fd2941161c0 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fd294116b70 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:52.437 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2940744a0 con 0x7fd294116b70 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd294117990 con 0x7fd294073910 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd294117b10 con 0x7fd2941161c0 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29bf98640 10 monclient(hunting): authenticate will time out at 1025.058199s 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd299d0d640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd294073910 0x7fd294073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd2941161c0 0x7fd294116630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd2941161c0 0x7fd294116630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.140:57808/0 (socket says 10.20.193.140:57808) 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 1 -- 10.20.193.140:0/113865489 learned_addr learned my addr 10.20.193.140:0/113865489 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 10 monclient(hunting): get_auth_request con 0x7fd2941161c0 auth_method 0 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.431+0000 7fd29950c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): get_auth_request con 0x7fd294073910 auth_method 0 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.438 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.432+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 1 -- 10.20.193.140:0/113865489 >> v2:10.20.193.68:3300/0 conn(0x7fd294073910 msgr2=0x7fd294073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 1 --2- 10.20.193.140:0/113865489 >> v2:10.20.193.68:3300/0 conn(0x7fd294073910 0x7fd294073cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 1 -- 10.20.193.140:0/113865489 >> v1:10.20.193.68:6789/0 conn(0x7fd294116b70 legacy=0x7fd294117280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 1 monclient: found mon.noname-c 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:52.439 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.433+0000 7fd29950c640 1 -- 10.20.193.140:0/113865489 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd294117cc0 con 0x7fd2941161c0 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.434+0000 7fd29950c640 1 --2- 10.20.193.140:0/113865489 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2941161c0 0x7fd294116630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd2940742f0 tx=0x7fd28802ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b87db091a65dd8 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 1 -- 10.20.193.140:0/113865489 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28803c070 con 0x7fd2941161c0 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: dump: 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.442 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 1 -- 10.20.193.140:0/113865489 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28802fb30 con 0x7fd2941161c0 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd29bf98640 5 monclient: authenticate success, global_id 4187 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 1 -- 10.20.193.140:0/113865489 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288037440 con 0x7fd2941161c0 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd29bf98640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd29bf98640 4 set_mon_vals no callback set 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.435+0000 7fd298d0b640 10 monclient: dump: 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.443 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: shutdown 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- 10.20.193.140:0/113865489 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2941161c0 msgr2=0x7fd294116630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- 10.20.193.140:0/113865489 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2941161c0 0x7fd294116630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd2940742f0 tx=0x7fd28802ee60 comp rx=0 tx=0).stop 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- 10.20.193.140:0/113865489 shutdown_connections 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- 10.20.193.140:0/113865489 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2941161c0 0x7fd294116630 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- 10.20.193.140:0/113865489 >> v2:10.20.193.68:3300/0 conn(0x7fd294073910 0x7fd294073cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- 10.20.193.140:0/113865489 >> 10.20.193.140:0/113865489 conn(0x7fd29406f5f0 msgr2=0x7fd294071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- 10.20.193.140:0/113865489 shutdown_connections 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- 10.20.193.140:0/113865489 wait complete. 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: monmap: 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:52.444 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 librados: starting msgr at 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 librados: starting objecter 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 Processor -- start 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- start start 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 librados: setting wanted keys 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 librados: calling monclient init 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: init 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 0x7fd2941b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-c con 0x7fd294073910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 0x7fd2941b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-b con 0x7fd2941161c0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:52.445 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd294116b70 0x7fd2941afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): picked mon.noname-a con 0x7fd294116b70 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd2941183d0 con 0x7fd294116b70 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd294118250 con 0x7fd2941161c0 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd294118550 con 0x7fd294073910 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.436+0000 7fd29bf98640 10 monclient(hunting): authenticate will time out at 1025.063784s 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 0x7fd2941b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 0x7fd2941b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 0x7fd2941b5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.140:57822/0 (socket says 10.20.193.140:57822) 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 1 -- 10.20.193.140:0/4178103964 learned_addr learned my addr 10.20.193.140:0/4178103964 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): get_auth_request con 0x7fd294073910 auth_method 0 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): get_auth_request con 0x7fd2941161c0 auth_method 0 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.446 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd299d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.447 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.437+0000 7fd29950c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.448 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 10 monclient(hunting): handle_auth_done global_id 4180 payload 824 2026-02-22T21:35:52.448 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 -- 10.20.193.140:0/4178103964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 msgr2=0x7fd2941b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 0x7fd2941b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 -- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd294116b70 msgr2=0x7fd2941afcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd294116b70 0x7fd2941afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 monclient: found mon.noname-b 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.438+0000 7fd29950c640 1 -- 10.20.193.140:0/4178103964 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2941b0590 con 0x7fd2941161c0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.439+0000 7fd29950c640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 0x7fd2941b6290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd294117c90 tx=0x7fd2880047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.439+0000 7fd2827fc640 10 client.?.objecter ms_handle_connect 0x7fd2941161c0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.439+0000 7fd2827fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28803c040 con 0x7fd2941161c0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: dump: 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.449 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 5 monclient: authenticate success, global_id 4180 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd288042670 con 0x7fd2941161c0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 10 monclient: _renew_subs 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2941b07c0 con 0x7fd2941161c0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd283fff640 4 set_mon_vals no callback set 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288042a60 con 0x7fd2941161c0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd2827fc640 10 monclient: dump: 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.450 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 10 monclient: _renew_subs 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2941bc9d0 con 0x7fd2941161c0 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.441+0000 7fd29bf98640 1 librados: init done 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.442+0000 7fd29bf98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.442+0000 7fd29bf98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.442+0000 7fd29bf98640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.451 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.442+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd294073cf0 con 0x7fd2941161c0 2026-02-22T21:35:52.457 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.453+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 3) ==== 99739+0+0 (secure 0 0 0) 0x7fd28802fce0 con 0x7fd2941161c0 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 0x7fd26407b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd299d0d640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 0x7fd26407b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd299d0d640 10 monclient: get_auth_request con 0x7fd264078ca0 auth_method 0 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2335+0+0 (secure 0 0 0) 0x7fd288047070 con 0x7fd2941161c0 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 10 client.4180.objecter ms_dispatch2 0x7fd294000b90 osd_map(16..16 src has 1..16) 2026-02-22T21:35:52.458 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 3 client.4180.objecter handle_osd_map got epochs [16,16] > 0 2026-02-22T21:35:52.459 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 3 client.4180.objecter handle_osd_map decoding full epoch 16 2026-02-22T21:35:52.459 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 20 client.4180.objecter dump_active .. 0 homeless 2026-02-22T21:35:52.459 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd299d0d640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 0x7fd26407b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd284009d00 tx=0x7fd284009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.459 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.454+0000 7fd2827fc640 10 client.4180.objecter ms_handle_connect 0x7fd264078ca0 2026-02-22T21:35:52.558 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.554+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd288086d20 con 0x7fd2941161c0 2026-02-22T21:35:52.559 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.554+0000 7fd2827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.559 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.554+0000 7fd2827fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:52.624 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.620+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7fd288082660 con 0x7fd2941161c0 2026-02-22T21:35:52.649 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.644+0000 7fd29bf98640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ce0888c2-dcdf-44c5-a8d4-06a1cd783264", "id": 14}] 2026-02-22T21:35:52.649 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.644+0000 7fd29bf98640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ce0888c2-dcdf-44c5-a8d4-06a1cd783264", "id": 14}] 2026-02-22T21:35:52.649 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.644+0000 7fd29bf98640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:52.649 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.644+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ce0888c2-dcdf-44c5-a8d4-06a1cd783264", "id": 14} v 0) -- 0x7fd2941bcc00 con 0x7fd2941161c0 2026-02-22T21:35:52.696 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.693+0000 7fd2827fc640 1 -- 10.20.193.140:0/4178103964 <== mon.1 v2:10.20.193.68:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce0888c2-dcdf-44c5-a8d4-06a1cd783264", "id": 14}]=0 v17) ==== 113+0+3 (secure 0 0 0) 0x7fd28808ff70 con 0x7fd2941161c0 2026-02-22T21:35:52.697 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.693+0000 7fd2827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ce0888c2-dcdf-44c5-a8d4-06a1cd783264", "id": 14}] 2026-02-22T21:35:52.697 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.693+0000 7fd2827fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:52.697 INFO:teuthology.orchestra.run.trial140.stdout:14 2026-02-22T21:35:52.699 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 10 librados: watch_flush enter 2026-02-22T21:35:52.699 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 10 librados: watch_flush exit 2026-02-22T21:35:52.699 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 20 client.4180.objecter shutdown clearing up homeless session... 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 10 client.4180.objecter successfully canceled tick 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 msgr2=0x7fd26407b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 0x7fd26407b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd284009d00 tx=0x7fd284009290 comp rx=0 tx=0).stop 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 10 monclient: shutdown 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 msgr2=0x7fd2941b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 0x7fd2941b6290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd294117c90 tx=0x7fd2880047c0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 shutdown_connections 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd264078ca0 0x7fd26407b160 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd294116b70 0x7fd2941afcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2941161c0 0x7fd2941b6290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.700 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 --2- 10.20.193.140:0/4178103964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd294073910 0x7fd2941b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.701 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 >> 10.20.193.140:0/4178103964 conn(0x7fd29406f5f0 msgr2=0x7fd294114580 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.701 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 shutdown_connections 2026-02-22T21:35:52.701 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 -- 10.20.193.140:0/4178103964 wait complete. 2026-02-22T21:35:52.701 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.694+0000 7fd29bf98640 1 librados: shutdown 2026-02-22T21:35:52.709 DEBUG:teuthology.orchestra.run.trial140:> sudo ceph --cluster ceph osd new 44725141-dc07-4bce-b2c5-121a3c85c67e 15 2026-02-22T21:35:52.772 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.768+0000 7f6a8eeb0640 10 monclient: get_monmap_and_config 2026-02-22T21:35:52.772 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.768+0000 7f6a8eeb0640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.768+0000 7f6a8eeb0640 10 monclient: monmap: 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:35:52.770366+0000 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:35:52.770366+0000 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 Processor -- start 2026-02-22T21:35:52.773 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 -- start start 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient: init 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient: _add_conns ranks=[4,0,5,3,2,1] 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 0x7f6a881127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-b con 0x7f6a881123f0 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a881172b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6a88117e70 0x7f6a8811f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-c con 0x7f6a88117e70 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8811f8b0 con 0x7f6a881172b0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f6a88120a90 con 0x7f6a881123f0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6a88120c10 con 0x7f6a88117e70 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.769+0000 7f6a8eeb0640 10 monclient(hunting): authenticate will time out at 1025.396681s 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 0x7f6a881127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6a88117e70 0x7f6a8811f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a7ffff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f6a881172b0 0x7f6a88117760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.140:48312/0 (socket says 10.20.193.140:48312) 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a7ffff640 1 -- 10.20.193.140:0/1502145917 learned_addr learned my addr 10.20.193.140:0/1502145917 (peer_addr_for_me v1:10.20.193.140:0/0) 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): get_auth_request con 0x7f6a881123f0 auth_method 0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): get_auth_request con 0x7f6a88117e70 auth_method 0 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.774 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.770+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 1 -- 10.20.193.140:0/1502145917 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097499267 0 0) 0x7f6a8811f8b0 con 0x7f6a881172b0 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 10 monclient(hunting): my global_id is 4194 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a7f7fe640 1 -- 10.20.193.140:0/1502145917 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a74003620 con 0x7f6a881172b0 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 1 -- 10.20.193.140:0/1502145917 >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 msgr2=0x7f6a881127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 1 --2- 10.20.193.140:0/1502145917 >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 0x7f6a881127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 1 -- 10.20.193.140:0/1502145917 >> v1:10.20.193.51:6789/0 conn(0x7f6a881172b0 legacy=0x7f6a88117760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 1 monclient: found mon.noname-c 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:52.775 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8d426640 1 -- 10.20.193.140:0/1502145917 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a88120d90 con 0x7f6a88117e70 2026-02-22T21:35:52.776 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.771+0000 7f6a8cc25640 1 --2- 10.20.193.140:0/1502145917 >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 0x7f6a881127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:35:52.776 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.772+0000 7f6a8d426640 1 --2- 10.20.193.140:0/1502145917 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a88117e70 0x7f6a8811f370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a88112e90 tx=0x7f6a8402f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5053b5528dfcfa57 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 1 -- 10.20.193.140:0/1502145917 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8403c070 con 0x7f6a88117e70 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 10 monclient: dump: 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.777 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a8eeb0640 5 monclient: authenticate success, global_id 4193 2026-02-22T21:35:52.778 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.773+0000 7f6a8eeb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 1 -- 10.20.193.140:0/1502145917 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8402fe40 con 0x7f6a88117e70 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 1 -- 10.20.193.140:0/1502145917 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84037640 con 0x7f6a88117e70 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a7f7fe640 10 monclient: dump: 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 4 set_mon_vals no callback set 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 10 monclient: shutdown 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1502145917 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a88117e70 msgr2=0x7f6a8811f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1502145917 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a88117e70 0x7f6a8811f370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a88112e90 tx=0x7f6a8402f130 comp rx=0 tx=0).stop 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1502145917 shutdown_connections 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1502145917 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a88117e70 0x7f6a8811f370 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1502145917 >> v2:10.20.193.68:3300/0 conn(0x7f6a881123f0 0x7f6a881127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1502145917 >> 10.20.193.140:0/1502145917 conn(0x7f6a8806f5f0 msgr2=0x7f6a88071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1502145917 shutdown_connections 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1502145917 wait complete. 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 10 monclient: build_initial_monmap 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 10 monclient: monmap: 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:epoch 0 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 0.000000 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:created 0.000000 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:52.779 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 librados: starting msgr at 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.774+0000 7f6a8eeb0640 1 librados: starting objecter 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 Processor -- start 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 -- start start 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 librados: setting wanted keys 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 librados: calling monclient init 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient: init 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-a con 0x7f6a881123f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a881172b0 0x7f6a88074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-c con 0x7f6a881172b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a88117e70 0x7f6a88074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): picked mon.noname-b con 0x7f6a88117e70 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8811ff40 con 0x7f6a881123f0 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8811fdc0 con 0x7f6a88117e70 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6a881200c0 con 0x7f6a881172b0 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:52.780 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8eeb0640 10 monclient(hunting): authenticate will time out at 1025.402214s 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8d426640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a88117e70 0x7f6a88074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a881172b0 0x7f6a88074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.140:52294/0 (socket says 10.20.193.140:52294) 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 1 -- 10.20.193.140:0/1999990870 learned_addr learned my addr 10.20.193.140:0/1999990870 (peer_addr_for_me v2:10.20.193.140:0/0) 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): get_auth_request con 0x7f6a881123f0 auth_method 0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): get_auth_request con 0x7f6a881172b0 auth_method 0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8d426640 10 monclient(hunting): get_auth_request con 0x7f6a88117e70 auth_method 0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8d426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8d426640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8d426640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.775+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.776+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.776+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.776+0000 7f6a8d426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 -- 10.20.193.140:0/1999990870 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a881172b0 msgr2=0x7f6a88074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a881172b0 0x7f6a88074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 -- 10.20.193.140:0/1999990870 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a88117e70 msgr2=0x7f6a88074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a88117e70 0x7f6a88074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 10 monclient: _finish_hunting 0 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 monclient: found mon.noname-a 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:52.781 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.782 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.777+0000 7f6a8cc25640 1 -- 10.20.193.140:0/1999990870 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a881b7960 con 0x7f6a881123f0 2026-02-22T21:35:52.782 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.778+0000 7f6a8cc25640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800ebe0 tx=0x7f6a7800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.782 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.778+0000 7f6a7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6a881123f0 2026-02-22T21:35:52.782 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.778+0000 7f6a7d7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:52.783 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78017070 con 0x7f6a881123f0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: dump: 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: _finish_auth 0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7800cc40 con 0x7f6a881123f0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78013470 con 0x7f6a881123f0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7effd640 4 set_mon_vals no callback set 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a7d7fa640 10 monclient: dump: 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:epoch 1 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:election_strategy: 1 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 5 monclient: authenticate success, global_id 4197 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _renew_subs 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881b7bf0 con 0x7f6a881123f0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _renew_subs 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a881b80f0 con 0x7f6a881123f0 2026-02-22T21:35:52.784 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 1 librados: init done 2026-02-22T21:35:52.785 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.785 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.785 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.785 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.780+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a881127d0 con 0x7f6a881123f0 2026-02-22T21:35:52.795 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.791+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f6a78024060 con 0x7f6a881123f0 2026-02-22T21:35:52.796 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.792+0000 7f6a7d7fa640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 0x7f6a5807b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:52.796 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.792+0000 7f6a7ffff640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 0x7f6a5807b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:52.796 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.792+0000 7f6a7ffff640 10 monclient: get_auth_request con 0x7f6a58078d70 auth_method 0 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2450+0+0 (secure 0 0 0) 0x7f6a7805e160 con 0x7f6a881123f0 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 10 client.4197.objecter ms_dispatch2 0x7f6a88000b90 osd_map(17..17 src has 1..17) 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 3 client.4197.objecter handle_osd_map got epochs [17,17] > 0 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 3 client.4197.objecter handle_osd_map decoding full epoch 17 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 20 client.4197.objecter dump_active .. 0 homeless 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7ffff640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 0x7f6a5807b230 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a70007b60 tx=0x7f6a70007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:52.797 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.793+0000 7f6a7d7fa640 10 client.4197.objecter ms_handle_connect 0x7f6a58078d70 2026-02-22T21:35:52.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.891+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6a78062120 con 0x7f6a881123f0 2026-02-22T21:35:52.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.891+0000 7f6a7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:52.895 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.891+0000 7f6a7d7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:52.983 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.979+0000 7f6a8eeb0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "44725141-dc07-4bce-b2c5-121a3c85c67e", "id": 15}] 2026-02-22T21:35:52.984 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.979+0000 7f6a8eeb0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "44725141-dc07-4bce-b2c5-121a3c85c67e", "id": 15}] 2026-02-22T21:35:52.984 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.979+0000 7f6a8eeb0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:35:52.984 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.979+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd new", "uuid": "44725141-dc07-4bce-b2c5-121a3c85c67e", "id": 15} v 0) -- 0x7f6a88119400 con 0x7f6a881123f0 2026-02-22T21:35:52.999 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a7d7fa640 1 -- 10.20.193.140:0/1999990870 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "44725141-dc07-4bce-b2c5-121a3c85c67e", "id": 15}]=0 v18) ==== 113+0+3 (secure 0 0 0) 0x7f6a78066420 con 0x7f6a881123f0 2026-02-22T21:35:52.999 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "44725141-dc07-4bce-b2c5-121a3c85c67e", "id": 15}] 2026-02-22T21:35:52.999 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a7d7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:52.999 INFO:teuthology.orchestra.run.trial140.stdout:15 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 10 librados: watch_flush enter 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 10 librados: watch_flush exit 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 20 client.4197.objecter shutdown clearing up homeless session... 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 10 client.4197.objecter successfully canceled tick 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 msgr2=0x7f6a5807b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 0x7f6a5807b230 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6a70007b60 tx=0x7f6a70007ab0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 10 monclient: shutdown 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 msgr2=0x7f6a88073e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800ebe0 tx=0x7f6a7800c520 comp rx=0 tx=0).stop 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 shutdown_connections 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a58078d70 0x7f6a5807b230 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a88117e70 0x7f6a88074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.000 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a881172b0 0x7f6a88074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.001 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 --2- 10.20.193.140:0/1999990870 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a881123f0 0x7f6a88073e00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.001 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 >> 10.20.193.140:0/1999990870 conn(0x7f6a8806f5f0 msgr2=0x7f6a880725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:53.001 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 shutdown_connections 2026-02-22T21:35:53.001 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 -- 10.20.193.140:0/1999990870 wait complete. 2026-02-22T21:35:53.001 INFO:teuthology.orchestra.run.trial140.stderr:2026-02-22T21:35:52.996+0000 7f6a8eeb0640 1 librados: shutdown 2026-02-22T21:35:53.007 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.007 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T21:35:53.007 DEBUG:teuthology.orchestra.run.trial051:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2026-02-22T21:35:53.009 INFO:tasks.ceph.osd.0:Started 2026-02-22T21:35:53.009 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.009 INFO:tasks.ceph.osd.4:Restarting daemon 2026-02-22T21:35:53.009 DEBUG:teuthology.orchestra.run.trial051:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2026-02-22T21:35:53.010 INFO:tasks.ceph.osd.4:Started 2026-02-22T21:35:53.010 DEBUG:tasks.ceph_manager:running osd.8 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.8.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.010 INFO:tasks.ceph.osd.8:Restarting daemon 2026-02-22T21:35:53.010 DEBUG:teuthology.orchestra.run.trial051:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.8.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 8 2026-02-22T21:35:53.011 INFO:tasks.ceph.osd.8:Started 2026-02-22T21:35:53.012 DEBUG:tasks.ceph_manager:running osd.12 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.12.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.012 INFO:tasks.ceph.osd.12:Restarting daemon 2026-02-22T21:35:53.012 DEBUG:teuthology.orchestra.run.trial051:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.12.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 12 2026-02-22T21:35:53.013 INFO:tasks.ceph.osd.12:Started 2026-02-22T21:35:53.013 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.013 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T21:35:53.013 DEBUG:teuthology.orchestra.run.trial068:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2026-02-22T21:35:53.014 INFO:tasks.ceph.osd.1:Started 2026-02-22T21:35:53.014 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.014 INFO:tasks.ceph.osd.5:Restarting daemon 2026-02-22T21:35:53.014 DEBUG:teuthology.orchestra.run.trial068:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2026-02-22T21:35:53.015 INFO:tasks.ceph.osd.5:Started 2026-02-22T21:35:53.015 DEBUG:tasks.ceph_manager:running osd.9 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.9.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.015 INFO:tasks.ceph.osd.9:Restarting daemon 2026-02-22T21:35:53.015 DEBUG:teuthology.orchestra.run.trial068:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.9.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 9 2026-02-22T21:35:53.017 INFO:tasks.ceph.osd.9:Started 2026-02-22T21:35:53.017 DEBUG:tasks.ceph_manager:running osd.13 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.13.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.017 INFO:tasks.ceph.osd.13:Restarting daemon 2026-02-22T21:35:53.017 DEBUG:teuthology.orchestra.run.trial068:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.13.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 13 2026-02-22T21:35:53.046 INFO:tasks.ceph.osd.13:Started 2026-02-22T21:35:53.046 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.046 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T21:35:53.046 DEBUG:teuthology.orchestra.run.trial125:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2026-02-22T21:35:53.047 INFO:tasks.ceph.osd.2:Started 2026-02-22T21:35:53.047 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.047 INFO:tasks.ceph.osd.6:Restarting daemon 2026-02-22T21:35:53.047 DEBUG:teuthology.orchestra.run.trial125:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2026-02-22T21:35:53.049 INFO:tasks.ceph.osd.6:Started 2026-02-22T21:35:53.049 DEBUG:tasks.ceph_manager:running osd.10 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.10.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.049 INFO:tasks.ceph.osd.10:Restarting daemon 2026-02-22T21:35:53.049 DEBUG:teuthology.orchestra.run.trial125:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.10.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 10 2026-02-22T21:35:53.050 INFO:tasks.ceph.osd.10:Started 2026-02-22T21:35:53.050 DEBUG:tasks.ceph_manager:running osd.14 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.14.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.050 INFO:tasks.ceph.osd.14:Restarting daemon 2026-02-22T21:35:53.050 DEBUG:teuthology.orchestra.run.trial125:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.14.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 14 2026-02-22T21:35:53.051 INFO:tasks.ceph.osd.14:Started 2026-02-22T21:35:53.051 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.051 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-22T21:35:53.051 DEBUG:teuthology.orchestra.run.trial140:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2026-02-22T21:35:53.052 INFO:tasks.ceph.osd.3:Started 2026-02-22T21:35:53.053 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.053 INFO:tasks.ceph.osd.7:Restarting daemon 2026-02-22T21:35:53.053 DEBUG:teuthology.orchestra.run.trial140:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2026-02-22T21:35:53.054 INFO:tasks.ceph.osd.7:Started 2026-02-22T21:35:53.054 DEBUG:tasks.ceph_manager:running osd.11 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.11.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.054 INFO:tasks.ceph.osd.11:Restarting daemon 2026-02-22T21:35:53.054 DEBUG:teuthology.orchestra.run.trial140:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.11.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 11 2026-02-22T21:35:53.055 INFO:tasks.ceph.osd.11:Started 2026-02-22T21:35:53.055 DEBUG:tasks.ceph_manager:running osd.15 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--gen-suppressions=all', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.15.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2026-02-22T21:35:53.055 INFO:tasks.ceph.osd.15:Restarting daemon 2026-02-22T21:35:53.055 DEBUG:teuthology.orchestra.run.trial140:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.15.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 15 2026-02-22T21:35:53.057 INFO:tasks.ceph.osd.15:Started 2026-02-22T21:35:53.057 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 10 monclient: get_monmap_and_config 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 10 monclient: build_initial_monmap 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 10 monclient: monmap: 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:35:53.124776+0000 2026-02-22T21:35:53.125 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:35:53.124776+0000 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:53.126 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 1 Processor -- start 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 1 -- start start 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.124+0000 7f90d7b14640 10 monclient: init 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient: _add_conns ranks=[3,1,4,0,2,5] 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f90d01527b0 0x7f90d00c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-a con 0x7f90d01527b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f90d00bf800 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f90d00703b0 0x7f90d015f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-b con 0x7f90d00703b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90d015ffa0 con 0x7f90d00bf800 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f90d00be810 con 0x7f90d01527b0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f90d0152b90 con 0x7f90d00703b0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d7b14640 10 monclient(hunting): authenticate will time out at 1028.705094s 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f90d01527b0 0x7f90d00c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d608a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f90d00703b0 0x7f90d015f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f90d01527b0 0x7f90d00c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60378/0 (socket says 10.20.193.51:60378) 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 1 -- 10.20.193.51:0/3241245797 learned_addr learned my addr 10.20.193.51:0/3241245797 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 10 monclient(hunting): get_auth_request con 0x7f90d01527b0 auth_method 0 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.125+0000 7f90d5889640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): get_auth_request con 0x7f90d00703b0 auth_method 0 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.126+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 1 -- 10.20.193.51:0/3241245797 >> v2:10.20.193.68:3300/0 conn(0x7f90d00703b0 msgr2=0x7f90d015f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 1 --2- 10.20.193.51:0/3241245797 >> v2:10.20.193.68:3300/0 conn(0x7f90d00703b0 0x7f90d015f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 1 -- 10.20.193.51:0/3241245797 >> v1:10.20.193.68:6789/0 conn(0x7f90d00bf800 legacy=0x7f90d006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 10 monclient: _finish_hunting 0 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 1 monclient: found mon.noname-a 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.127+0000 7f90d5889640 1 -- 10.20.193.51:0/3241245797 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d00bea60 con 0x7f90d01527b0 2026-02-22T21:35:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.128+0000 7f90d5889640 1 --2- 10.20.193.51:0/3241245797 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d00c9660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90d0156280 tx=0x7f90c002eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59838b55305469a2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 1 -- 10.20.193.51:0/3241245797 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c003c070 con 0x7f90d01527b0 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: dump: 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:53.132 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: _finish_auth 0 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: _check_auth_tickets 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 1 -- 10.20.193.51:0/3241245797 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c002fbe0 con 0x7f90d01527b0 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d7b14640 5 monclient: authenticate success, global_id 4200 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d7b14640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 1 -- 10.20.193.51:0/3241245797 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c0037440 con 0x7f90d01527b0 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d4887640 10 monclient: dump: 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:53.133 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d7b14640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.130+0000 7f90d7b14640 4 set_mon_vals no callback set 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 10 monclient: shutdown 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- 10.20.193.51:0/3241245797 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 msgr2=0x7f90d00c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3241245797 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d00c9660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f90d0156280 tx=0x7f90c002eed0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- 10.20.193.51:0/3241245797 shutdown_connections 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3241245797 >> v2:10.20.193.68:3300/0 conn(0x7f90d00703b0 0x7f90d015f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3241245797 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d00c9660 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- 10.20.193.51:0/3241245797 >> 10.20.193.51:0/3241245797 conn(0x7f90d00cd7b0 msgr2=0x7f90d00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- 10.20.193.51:0/3241245797 shutdown_connections 2026-02-22T21:35:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- 10.20.193.51:0/3241245797 wait complete. 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 10 monclient: build_initial_monmap 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 10 monclient: monmap: 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:35:53.136 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 librados: starting msgr at 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 librados: starting objecter 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 Processor -- start 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 -- start start 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 librados: setting wanted keys 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 1 librados: calling monclient init 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.131+0000 7f90d7b14640 10 monclient: init 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f90d00703b0 0x7f90d00c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-c con 0x7f90d00703b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 0x7f90d015adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-b con 0x7f90d00bf800 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d01f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): picked mon.noname-a con 0x7f90d01527b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f90d00bf300 con 0x7f90d01527b0 2026-02-22T21:35:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f90d00bf020 con 0x7f90d00bf800 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f90d006d710 con 0x7f90d00703b0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d7b14640 10 monclient(hunting): authenticate will time out at 1028.712363s 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d01f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5088640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 0x7f90d015adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d01f5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60394/0 (socket says 10.20.193.51:60394) 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 1 -- 10.20.193.51:0/3821139513 learned_addr learned my addr 10.20.193.51:0/3821139513 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 10 monclient(hunting): get_auth_request con 0x7f90d01527b0 auth_method 0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d608a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5088640 10 monclient(hunting): get_auth_request con 0x7f90d00bf800 auth_method 0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5088640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5088640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5088640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.132+0000 7f90d5889640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f90d00703b0 0x7f90d00c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d608a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5088640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5088640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5088640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): get_auth_request con 0x7f90d00703b0 auth_method 0 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.133+0000 7f90d5889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 10 monclient(hunting): handle_auth_done global_id 4195 payload 824 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 -- 10.20.193.51:0/3821139513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f90d00703b0 msgr2=0x7f90d00c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f90d00703b0 0x7f90d00c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 -- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 msgr2=0x7f90d01f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d01f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 10 monclient: _finish_hunting 0 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 monclient: found mon.noname-b 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:35:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.134+0000 7f90d5088640 1 -- 10.20.193.51:0/3821139513 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d00c7100 con 0x7f90d00bf800 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.135+0000 7f90d5088640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 0x7f90d015adf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c460 tx=0x7f90c400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.135+0000 7f90be7fc640 10 client.?.objecter ms_handle_connect 0x7f90d00bf800 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.135+0000 7f90be7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.136+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c40052d0 con 0x7f90d00bf800 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.136+0000 7f90be7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: dump: 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: _finish_auth 0 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:35:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 5 monclient: authenticate success, global_id 4195 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _renew_subs 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d00c73e0 con 0x7f90d00bf800 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _renew_subs 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d01f5720 con 0x7f90d00bf800 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 1 librados: init done 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d00bf540 con 0x7f90d00bf800 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c4005470 con 0x7f90d00bf800 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c40038a0 con 0x7f90d00bf800 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90bffff640 4 set_mon_vals no callback set 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.137+0000 7f90be7fc640 10 monclient: dump: 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:53.142 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:53.143 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:35:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f90c4003a40 con 0x7f90d00bf800 2026-02-22T21:35:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90be7fc640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 0x7f90a007b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90d5889640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 0x7f90a007b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90d5889640 10 monclient: get_auth_request con 0x7f90a0078d70 auth_method 0 2026-02-22T21:35:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90be7fc640 10 client.4195.objecter ms_handle_connect 0x7f90a0078d70 2026-02-22T21:35:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.246+0000 7f90d5889640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 0x7f90a007b230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f90d00709e0 tx=0x7f90c0039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f90c4097770 con 0x7f90d00bf800 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 10 client.4195.objecter ms_dispatch2 0x7f90d0000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 3 client.4195.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 3 client.4195.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 20 client.4195.objecter dump_active .. 0 homeless 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f90d00bf540 con 0x7f90d00bf800 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:53.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.253+0000 7f90be7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:53.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.340+0000 7f90d7b14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:53.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.340+0000 7f90d7b14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:53.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.340+0000 7f90d7b14640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:35:53.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.340+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90d00c77f0 con 0x7f90d00bf800 2026-02-22T21:35:53.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.395+0000 7f90be7fc640 1 -- 10.20.193.51:0/3821139513 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f90c4061e90 con 0x7f90d00bf800 2026-02-22T21:35:53.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.395+0000 7f90be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:53.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.395+0000 7f90be7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:53.398 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:35:53.398 INFO:teuthology.orchestra.run.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:35:53.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 10 librados: watch_flush enter 2026-02-22T21:35:53.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 10 librados: watch_flush exit 2026-02-22T21:35:53.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 20 client.4195.objecter shutdown clearing up homeless session... 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 10 client.4195.objecter successfully canceled tick 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 msgr2=0x7f90a007b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 0x7f90a007b230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f90d00709e0 tx=0x7f90c0039040 comp rx=0 tx=0).stop 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 10 monclient: shutdown 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 msgr2=0x7f90d015adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 0x7f90d015adf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c460 tx=0x7f90c400c930 comp rx=0 tx=0).stop 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 shutdown_connections 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f90a0078d70 0x7f90a007b230 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90d01527b0 0x7f90d01f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90d00bf800 0x7f90d015adf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 --2- 10.20.193.51:0/3821139513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f90d00703b0 0x7f90d00c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:53.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 >> 10.20.193.51:0/3821139513 conn(0x7f90d00cd7b0 msgr2=0x7f90d00c8230 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:53.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 shutdown_connections 2026-02-22T21:35:53.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 -- 10.20.193.51:0/3821139513 wait complete. 2026-02-22T21:35:53.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:35:53.398+0000 7f90d7b14640 1 librados: shutdown 2026-02-22T21:35:53.409 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-22T21:35:53.409 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T21:35:55.021 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:35:55.082 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 10 monclient: get_monmap_and_config 2026-02-22T21:35:55.082 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 10 monclient: build_initial_monmap 2026-02-22T21:35:55.082 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 10 monclient: monmap: 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:35:55.081033+0000 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:35:55.081033+0000 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:35:55.083 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 1 Processor -- start 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 1 -- start start 2026-02-22T21:35:55.084 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.080+0000 7feb5d26d640 10 monclient: init 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient: _add_conns ranks=[5,1,0,2,4,3] 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7feb5806ff30 0x7feb58070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-c con 0x7feb5806ff30 addr v2:10.20.193.125:3300/0 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7feb580c92d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb580c9700 addr v1:10.20.193.51:6789/0 2026-02-22T21:35:55.092 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb580c5570 con 0x7feb580c9700 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb580c5290 con 0x7feb580c92d0 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7feb580c9b10 con 0x7feb5806ff30 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.081+0000 7feb5d26d640 10 monclient(hunting): authenticate will time out at 1030.661459s 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56575640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7feb580c92d0 0x7feb58159ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54486/0 (socket says 10.20.193.51:54486) 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56575640 1 -- 10.20.193.51:0/3315412115 learned_addr learned my addr 10.20.193.51:0/3315412115 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 1 --2- 10.20.193.51:0/3315412115 >> v2:10.20.193.125:3300/0 conn(0x7feb5806ff30 0x7feb58070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): get_auth_request con 0x7feb5806ff30 auth_method 0 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:55.093 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.082+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613446587 0 0) 0x7feb580c5290 con 0x7feb580c92d0 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _negotiate 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): my global_id is 4198 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb44003620 con 0x7feb580c92d0 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116383105 0 0) 0x7feb48002ca0 con 0x7feb580c9700 2026-02-22T21:35:55.094 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _negotiate 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 10 monclient(hunting): my global_id is 4206 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.083+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb580c5290 con 0x7feb580c9700 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 1 -- 10.20.193.51:0/3315412115 >> v1:10.20.193.68:6789/0 conn(0x7feb580c92d0 legacy=0x7feb58159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 1 -- 10.20.193.51:0/3315412115 >> v1:10.20.193.51:6789/0 conn(0x7feb580c9700 legacy=0x7feb580c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 10 monclient: _finish_hunting 0 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 1 monclient: found mon.noname-c 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.095 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.084+0000 7feb56d76640 1 -- 10.20.193.51:0/3315412115 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5815d1e0 con 0x7feb5806ff30 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.085+0000 7feb56d76640 1 --2- 10.20.193.51:0/3315412115 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb5806ff30 0x7feb58070310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0098a0 tx=0x7feb4c02ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aa69c68363475c3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c03c070 con 0x7feb5806ff30 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 10 monclient: dump: 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.096 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 10 monclient: _finish_auth 0 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 10 monclient: _check_auth_tickets 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb55d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb5d26d640 5 monclient: authenticate success, global_id 4202 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.086+0000 7feb5d26d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:35:55.097 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4c02fc10 con 0x7feb5806ff30 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 1 -- 10.20.193.51:0/3315412115 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c037440 con 0x7feb5806ff30 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb55d74640 10 monclient: dump: 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.098 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 10 monclient: get_monmap_and_config success 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 4 set_mon_vals no callback set 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 10 monclient: shutdown 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 -- 10.20.193.51:0/3315412115 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb5806ff30 msgr2=0x7feb58070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 --2- 10.20.193.51:0/3315412115 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb5806ff30 0x7feb58070310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0098a0 tx=0x7feb4c02ef00 comp rx=0 tx=0).stop 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 -- 10.20.193.51:0/3315412115 shutdown_connections 2026-02-22T21:35:55.099 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 --2- 10.20.193.51:0/3315412115 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb5806ff30 0x7feb58070310 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 -- 10.20.193.51:0/3315412115 >> 10.20.193.51:0/3315412115 conn(0x7feb580cd4f0 msgr2=0x7feb580cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 -- 10.20.193.51:0/3315412115 shutdown_connections 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 -- 10.20.193.51:0/3315412115 wait complete. 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 10 monclient: build_initial_monmap 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 10 monclient: monmap: 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:35:55.100 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 librados: starting msgr at 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.087+0000 7feb5d26d640 1 librados: starting objecter 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 Processor -- start 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 -- start start 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 librados: setting wanted keys 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 librados: calling monclient init 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient: init 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient: _reopen_session rank -1 2026-02-22T21:35:55.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feb5806ff30 0x7feb5806ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-a con 0x7feb5806ff30 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feb580c92d0 0x7feb581f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-b con 0x7feb580c92d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): picked mon.noname-c con 0x7feb580c9700 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7feb5815d7a0 con 0x7feb5806ff30 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7feb5815db10 con 0x7feb580c92d0 2026-02-22T21:35:55.102 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7feb5815a1d0 con 0x7feb580c9700 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): _renew_subs 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb5d26d640 10 monclient(hunting): authenticate will time out at 1030.668433s 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb56d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feb5806ff30 0x7feb5806ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb56575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feb580c92d0 0x7feb581f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:60662/0 (socket says 10.20.193.51:60662) 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 1 -- 10.20.193.51:0/2202334607 learned_addr learned my addr 10.20.193.51:0/2202334607 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 10 monclient(hunting): get_auth_request con 0x7feb580c9700 auth_method 0 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.088+0000 7feb57577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.103 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): get_auth_request con 0x7feb5806ff30 auth_method 0 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): get_auth_request con 0x7feb580c92d0 auth_method 0 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb57577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb57577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb57577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:55.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:55.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:35:55.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:35:55.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.089+0000 7feb56575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:35:55.107 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2026-02-22T21:35:55.107 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 -- 10.20.193.51:0/2202334607 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feb580c92d0 msgr2=0x7feb581f2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feb580c92d0 0x7feb581f2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 -- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feb5806ff30 msgr2=0x7feb5806ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feb5806ff30 0x7feb5806ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 10 monclient: _finish_hunting 0 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 monclient: found mon.noname-c 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.090+0000 7feb57577640 1 -- 10.20.193.51:0/2202334607 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb581f57d0 con 0x7feb580c9700 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.092+0000 7feb57577640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cc20 tx=0x7feb4800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:55.108 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.092+0000 7feb377fe640 10 client.?.objecter ms_handle_connect 0x7feb580c9700 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.092+0000 7feb377fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4800ce40 con 0x7feb580c9700 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: dump: 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.109 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: _finish_auth 0 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: _check_auth_tickets 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb48003a70 con 0x7feb580c9700 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb5d26d640 5 monclient: authenticate success, global_id 4205 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb5d26d640 10 monclient: _renew_subs 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb5d26d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb581f1830 con 0x7feb580c9700 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:35:55.110 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48003e60 con 0x7feb580c9700 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb377fe640 10 monclient: dump: 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:35:55.111 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.093+0000 7feb55573640 4 set_mon_vals no callback set 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 monclient: _renew_subs 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb581f1df0 con 0x7feb580c9700 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 1 librados: init done 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.094+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb58070310 con 0x7feb580c9700 2026-02-22T21:35:55.112 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.106+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7feb480050d0 con 0x7feb580c9700 2026-02-22T21:35:55.113 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.106+0000 7feb377fe640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 0x7feb2407b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:35:55.113 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.106+0000 7feb56d76640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 0x7feb2407b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:35:55.113 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.106+0000 7feb56d76640 10 monclient: get_auth_request con 0x7feb24078d70 auth_method 0 2026-02-22T21:35:55.113 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.107+0000 7feb377fe640 10 client.4205.objecter ms_handle_connect 0x7feb24078d70 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.107+0000 7feb56d76640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 0x7feb2407b230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb4c039000 tx=0x7feb4c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.108+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7feb4805e730 con 0x7feb580c9700 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.108+0000 7feb377fe640 10 client.4205.objecter ms_dispatch2 0x7feb58000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.108+0000 7feb377fe640 3 client.4205.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.108+0000 7feb377fe640 3 client.4205.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:35:55.114 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.108+0000 7feb377fe640 20 client.4205.objecter dump_active .. 0 homeless 2026-02-22T21:35:55.217 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.215+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7feb580c5570 con 0x7feb580c9700 2026-02-22T21:35:55.217 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.215+0000 7feb377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:35:55.218 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.215+0000 7feb377fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:35:55.303 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.301+0000 7feb5d26d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:55.304 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.301+0000 7feb5d26d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:55.304 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.301+0000 7feb5d26d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:35:55.304 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.301+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb58157370 con 0x7feb580c9700 2026-02-22T21:35:55.330 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.328+0000 7feb377fe640 1 -- 10.20.193.51:0/2202334607 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7feb480626f0 con 0x7feb580c9700 2026-02-22T21:35:55.331 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.328+0000 7feb377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:35:55.331 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.328+0000 7feb377fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:35:55.331 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:35:55.331 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 10 librados: watch_flush enter 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 10 librados: watch_flush exit 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 20 client.4205.objecter shutdown clearing up homeless session... 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 10 client.4205.objecter successfully canceled tick 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 msgr2=0x7feb2407b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 0x7feb2407b230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb4c039000 tx=0x7feb4c033000 comp rx=0 tx=0).stop 2026-02-22T21:35:55.334 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 10 monclient: shutdown 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 msgr2=0x7feb581f4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cc20 tx=0x7feb4800a430 comp rx=0 tx=0).stop 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 shutdown_connections 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feb24078d70 0x7feb2407b230 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feb580c9700 0x7feb581f4c90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feb580c92d0 0x7feb581f2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 --2- 10.20.193.51:0/2202334607 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feb5806ff30 0x7feb5806ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 >> 10.20.193.51:0/2202334607 conn(0x7feb580cd4f0 msgr2=0x7feb580cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 shutdown_connections 2026-02-22T21:35:55.335 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 -- 10.20.193.51:0/2202334607 wait complete. 2026-02-22T21:35:55.336 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:35:55.331+0000 7feb5d26d640 1 librados: shutdown 2026-02-22T21:35:55.342 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:02.849 INFO:tasks.ceph.osd.15.trial140.stderr:--00:00:00:09.741 45037-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.849 INFO:tasks.ceph.osd.3.trial140.stderr:--00:00:00:09.742 45034-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.850 INFO:tasks.ceph.osd.7.trial140.stderr:--00:00:00:09.742 45036-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.851 INFO:tasks.ceph.osd.11.trial140.stderr:--00:00:00:09.743 45035-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.854 INFO:tasks.ceph.osd.4.trial051.stderr:--00:00:00:09.797 45072-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.854 INFO:tasks.ceph.osd.0.trial051.stderr:--00:00:00:09.798 45070-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.855 INFO:tasks.ceph.osd.12.trial051.stderr:--00:00:00:09.795 45073-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.857 INFO:tasks.ceph.osd.8.trial051.stderr:--00:00:00:09.800 45071-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.870 INFO:tasks.ceph.osd.10.trial125.stderr:--00:00:00:09.774 44670-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.871 INFO:tasks.ceph.osd.6.trial125.stderr:--00:00:00:09.776 44669-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.871 INFO:tasks.ceph.osd.2.trial125.stderr:--00:00:00:09.776 44668-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.872 INFO:tasks.ceph.osd.14.trial125.stderr:--00:00:00:09.776 44671-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.902 INFO:tasks.ceph.osd.13.trial068.stderr:--00:00:00:09.811 44662-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.902 INFO:tasks.ceph.osd.1.trial068.stderr:--00:00:00:09.839 44656-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.903 INFO:tasks.ceph.osd.5.trial068.stderr:--00:00:00:09.842 44652-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.903 INFO:tasks.ceph.osd.9.trial068.stderr:--00:00:00:09.839 44655-- Warning: zero subprog, missing DW_AT_abstract_origin in DW_TAG_inlined_subroutine in /usr/lib/debug/.build-id/8c/c8adc374e188776041495b4f4f83fb04e98a9a.debug 2026-02-22T21:36:02.954 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.010+0000 7f4376d8f640 10 monclient: get_monmap_and_config 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.010+0000 7f4376d8f640 10 monclient: build_initial_monmap 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.010+0000 7f4376d8f640 10 monclient: monmap: 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:03.011398+0000 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:03.011398+0000 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:03.012 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 Processor -- start 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 -- start start 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient: init 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient: _add_conns ranks=[5,2,4,1,3,0] 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f437006ff30 0x7f4370070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f437006ff30 addr v2:10.20.193.125:3300/0 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43700c92d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f43700c9700 0x7f43700c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f43700c9700 addr v2:10.20.193.68:3300/0 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43700c5570 con 0x7f43700c92d0 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f43700c9b10 con 0x7f43700c9700 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f437015db10 con 0x7f437006ff30 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.011+0000 7f4376d8f640 10 monclient(hunting): authenticate will time out at 1038.591694s 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4374b04640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f437006ff30 0x7f4370070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.013 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f43700c9700 0x7f43700c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f43700c9700 0x7f43700c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35836/0 (socket says 10.20.193.51:35836) 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 1 -- 10.20.193.51:0/2835723896 learned_addr learned my addr 10.20.193.51:0/2835723896 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): get_auth_request con 0x7f43700c9700 auth_method 0 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4374b04640 10 monclient(hunting): get_auth_request con 0x7f437006ff30 auth_method 0 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4374b04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4374b04640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4374b04640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.012+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:03.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4375305640 10 monclient(hunting): handle_auth_done global_id 4204 payload 274 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4375305640 1 -- 10.20.193.51:0/2835723896 >> v2:10.20.193.125:3300/0 conn(0x7f437006ff30 msgr2=0x7f4370070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4375305640 1 --2- 10.20.193.51:0/2835723896 >> v2:10.20.193.125:3300/0 conn(0x7f437006ff30 0x7f4370070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.013+0000 7f4375305640 1 -- 10.20.193.51:0/2835723896 >> v1:10.20.193.125:6789/0 conn(0x7f43700c92d0 legacy=0x7f4370159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.014+0000 7f4375305640 10 monclient: _finish_hunting 0 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.014+0000 7f4375305640 1 monclient: found mon.noname-b 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.014+0000 7f4375305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.014+0000 7f4375305640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.015 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.014+0000 7f4375305640 1 -- 10.20.193.51:0/2835723896 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43700c5290 con 0x7f43700c9700 2026-02-22T21:36:03.016 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.015+0000 7f4375305640 1 --2- 10.20.193.51:0/2835723896 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c9700 0x7f43700c4710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43700ce2f0 tx=0x7f436c02f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1fa6b7b179ce8c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 1 -- 10.20.193.51:0/2835723896 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c02fe10 con 0x7f43700c9700 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: dump: 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: _finish_auth 0 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: _check_auth_tickets 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 1 -- 10.20.193.51:0/2835723896 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f436c0373f0 con 0x7f43700c9700 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 1 -- 10.20.193.51:0/2835723896 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c0377c0 con 0x7f43700c9700 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 5 monclient: authenticate success, global_id 4204 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 4 set_mon_vals no callback set 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:03.018 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f43677fe640 10 monclient: dump: 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 10 monclient: shutdown 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 -- 10.20.193.51:0/2835723896 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c9700 msgr2=0x7f43700c4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 --2- 10.20.193.51:0/2835723896 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c9700 0x7f43700c4710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43700ce2f0 tx=0x7f436c02f310 comp rx=0 tx=0).stop 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 -- 10.20.193.51:0/2835723896 shutdown_connections 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 --2- 10.20.193.51:0/2835723896 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c9700 0x7f43700c4710 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 --2- 10.20.193.51:0/2835723896 >> v2:10.20.193.125:3300/0 conn(0x7f437006ff30 0x7f4370070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 -- 10.20.193.51:0/2835723896 >> 10.20.193.51:0/2835723896 conn(0x7f43700cd4f0 msgr2=0x7f43700cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.016+0000 7f4376d8f640 1 -- 10.20.193.51:0/2835723896 shutdown_connections 2026-02-22T21:36:03.019 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 -- 10.20.193.51:0/2835723896 wait complete. 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 10 monclient: build_initial_monmap 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 10 monclient: monmap: 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 librados: starting msgr at 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 librados: starting objecter 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 Processor -- start 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 -- start start 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 librados: setting wanted keys 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 1 librados: calling monclient init 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 10 monclient: init 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.017+0000 7f4376d8f640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f437006ff30 0x7f43700cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f437006ff30 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:03.020 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f43700c92d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f43700c9700 0x7f43700bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f43700c9700 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f437015d7a0 con 0x7f437006ff30 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f437015d4c0 con 0x7f43700c92d0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f437015a1d0 con 0x7f43700c9700 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4376d8f640 10 monclient(hunting): authenticate will time out at 1038.598049s 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f43700c9700 0x7f43700bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35838/0 (socket says 10.20.193.51:35838) 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4374b04640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f437006ff30 0x7f43700cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 1 -- 10.20.193.51:0/3777756736 learned_addr learned my addr 10.20.193.51:0/3777756736 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): get_auth_request con 0x7f43700c92d0 auth_method 0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): get_auth_request con 0x7f43700c9700 auth_method 0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4374b04640 10 monclient(hunting): get_auth_request con 0x7f437006ff30 auth_method 0 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4374b04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4374b04640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4374b04640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:03.021 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.018+0000 7f4375305640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.019+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.019+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.019+0000 7f4374b04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 -- 10.20.193.51:0/3777756736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f43700c9700 msgr2=0x7f43700bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f43700c9700 0x7f43700bf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 -- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f437006ff30 msgr2=0x7f43700cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f437006ff30 0x7f43700cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 10 monclient: _finish_hunting 0 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 monclient: found mon.noname-b 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.020+0000 7f4367fff640 1 -- 10.20.193.51:0/3777756736 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43700be8a0 con 0x7f43700c92d0 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.021+0000 7f4367fff640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f435800c580 tx=0x7f435800ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.021+0000 7f43657fa640 10 client.?.objecter ms_handle_connect 0x7f43700c92d0 2026-02-22T21:36:03.022 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.021+0000 7f43657fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358017070 con 0x7f43700c92d0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: dump: 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: _finish_auth 0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: _check_auth_tickets 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43580052d0 con 0x7f43700c92d0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43580036b0 con 0x7f43700c92d0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 5 monclient: authenticate success, global_id 4207 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 10 monclient: _renew_subs 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437015c190 con 0x7f43700c92d0 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4366ffd640 4 set_mon_vals no callback set 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f43657fa640 10 monclient: dump: 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 10 monclient: _renew_subs 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43700beb40 con 0x7f43700c92d0 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.022+0000 7f4376d8f640 1 librados: init done 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.023+0000 7f4376d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.023+0000 7f4376d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.023+0000 7f4376d8f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.025 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.023+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4370157470 con 0x7f43700c92d0 2026-02-22T21:36:03.035 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.034+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f4358003850 con 0x7f43700c92d0 2026-02-22T21:36:03.036 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.034+0000 7f43657fa640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 0x7f434007b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:03.036 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f4374b04640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 0x7f434007b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:03.036 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f4374b04640 10 monclient: get_auth_request con 0x7f4340078dc0 auth_method 0 2026-02-22T21:36:03.036 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 10 client.4207.objecter ms_handle_connect 0x7f4340078dc0 2026-02-22T21:36:03.036 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f4374b04640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 0x7f434007b280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4360009c40 tx=0x7f4360009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:03.037 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f435805d3d0 con 0x7f43700c92d0 2026-02-22T21:36:03.037 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 10 client.4207.objecter ms_dispatch2 0x7f4370000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:03.037 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 3 client.4207.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:03.037 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 3 client.4207.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:03.037 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.035+0000 7f43657fa640 20 client.4207.objecter dump_active .. 0 homeless 2026-02-22T21:36:03.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.139+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f4358021020 con 0x7f43700c92d0 2026-02-22T21:36:03.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.139+0000 7f43657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:03.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.139+0000 7f43657fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:03.227 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.226+0000 7f4376d8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:03.227 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.226+0000 7f4376d8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:03.228 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.226+0000 7f4376d8f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:03.228 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.226+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43700cda50 con 0x7f43700c92d0 2026-02-22T21:36:03.240 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.238+0000 7f43657fa640 1 -- 10.20.193.51:0/3777756736 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f43700c5570 con 0x7f43700c92d0 2026-02-22T21:36:03.240 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.238+0000 7f43657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:03.240 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.238+0000 7f43657fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:03.240 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:03.240 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:03.242 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 10 librados: watch_flush enter 2026-02-22T21:36:03.242 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 10 librados: watch_flush exit 2026-02-22T21:36:03.242 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 20 client.4207.objecter shutdown clearing up homeless session... 2026-02-22T21:36:03.242 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 10 client.4207.objecter successfully canceled tick 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 msgr2=0x7f434007b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 0x7f434007b280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4360009c40 tx=0x7f4360009290 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 10 monclient: shutdown 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 msgr2=0x7f437015b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f435800c580 tx=0x7f435800ca50 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 shutdown_connections 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4340078dc0 0x7f434007b280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f43700c9700 0x7f43700bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f43700c92d0 0x7f437015b850 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 --2- 10.20.193.51:0/3777756736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f437006ff30 0x7f43700cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 >> 10.20.193.51:0/3777756736 conn(0x7f43700cd4f0 msgr2=0x7f437014a510 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 shutdown_connections 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 -- 10.20.193.51:0/3777756736 wait complete. 2026-02-22T21:36:03.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:03.241+0000 7f4376d8f640 1 librados: shutdown 2026-02-22T21:36:03.250 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:07.348 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.239 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.348 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:14.240 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.371 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:14.263 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.372 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:14.264 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.385 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:14.290 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.386 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:14.289 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.386 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:14.291 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.414 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.357 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.426 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:14.329 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.445 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:14.387 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.463 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:14.400 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.463 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:14.372 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.464 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:14.400 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.464 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:14.403 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.466 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:14.410 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.474 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:14.414 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.641 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.533 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.650 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:14.542 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.665 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:14.557 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.666 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.558 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.669 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:14.561 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.671 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:14.577 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.672 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:14.576 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.676 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:14.581 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.686 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.578 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.696 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:14.601 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.699 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.643 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.700 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:14.604 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.720 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:14.623 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.724 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.667 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.731 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:14.674 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.745 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:14.648 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.746 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.690 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.757 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:14.694 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.758 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:14.701 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.761 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:14.698 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.761 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:14.701 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.765 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:14.674 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.765 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:14.708 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.769 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:14.710 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.779 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:14.722 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.786 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:14.723 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.787 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:14.728 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.788 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:14.731 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.794 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:14.734 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.794 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:14.704 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.810 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:14.748 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.818 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:14.728 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.823 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:14.764 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.913 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:14.805 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.939 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:14.831 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.939 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:14.831 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.940 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:14.845 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.943 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:14.835 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.963 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.855 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.964 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:14.869 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.964 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:14.868 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.972 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:14.865 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:07.990 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.882 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.005 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:14.897 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.008 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:14.911 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.010 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.953 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.024 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:14.962 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.031 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.974 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.046 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:14.989 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.046 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:14.988 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.054 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:14.997 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.057 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:14.997 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.084 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:15.021 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.088 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:14.997 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.092 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:15.032 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.114 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:15.023 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.129 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:15.039 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.204 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:15.109 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.205 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:15.098 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.209 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:15.102 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.231 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:15.134 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.231 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:15.135 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.244 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:15.135 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.252 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:15.156 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.268 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:15.171 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.275 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:15.178 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.289 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:15.227 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.312 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:15.254 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.322 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:15.261 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.327 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:15.270 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.338 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:15.281 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.353 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:15.289 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.364 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:15.305 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.368 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:15.305 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.380 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:15.320 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.468 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:15.373 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.474 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:15.366 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.476 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:15.368 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.489 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:15.382 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.503 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:15.408 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.518 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:15.423 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.524 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:15.416 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.541 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:36:08.534+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.541 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:15.434 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.551 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:15.454 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.553 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:15.490 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.567 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:15.470 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.574 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:36:08.570+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.586 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:15.526 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.602 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:15.542 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.617 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:15.560 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.633 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:15.576 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.662 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:36:08.654+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.736 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:15.641 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.743 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:15.635 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.751 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:15.656 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.759 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:15.651 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.804 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:36:08.799+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.829 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:15.766 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.844 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:15.782 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:08.859 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:36:08.855+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.913 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:36:08.905+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:08.930 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:36:08.925+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.018 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:36:09.011+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.047 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:36:09.043+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.086 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:36:09.079+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.105 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:36:09.100+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.137 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:36:09.129+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.165 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:36:09.161+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.296 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:36:09.288+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.299 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:36:09.294+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:09.388 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:36:09.380+0000 77b8940 -1 Falling back to public interface 2026-02-22T21:36:10.861 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:10.921 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 10 monclient: get_monmap_and_config 2026-02-22T21:36:10.921 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 10 monclient: build_initial_monmap 2026-02-22T21:36:10.921 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 10 monclient: monmap: 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:10.920170+0000 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:10.920170+0000 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:10.922 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:10.923 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:10.923 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 1 Processor -- start 2026-02-22T21:36:10.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 1 -- start start 2026-02-22T21:36:10.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.919+0000 7fae671c3640 10 monclient: init 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient: _add_conns ranks=[4,2,5,1,0,3] 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fae6015f480 0x7fae6015f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-b con 0x7fae6015f480 addr v2:10.20.193.68:3300/0 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae600c84e0 addr v1:10.20.193.125:6789/0 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fae60151ff0 0x7fae60152460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-c con 0x7fae60151ff0 addr v2:10.20.193.125:3300/0 2026-02-22T21:36:10.926 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae600c8930 con 0x7fae600c84e0 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fae600c9f50 con 0x7fae6015f480 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fae600c8bd0 con 0x7fae60151ff0 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.920+0000 7fae671c3640 10 monclient(hunting): authenticate will time out at 1046.500738s 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fae6015f480 0x7fae6015f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae65739640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fae60151ff0 0x7fae60152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fae6015f480 0x7fae6015f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:32958/0 (socket says 10.20.193.51:32958) 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 1 -- 10.20.193.51:0/1456544149 learned_addr learned my addr 10.20.193.51:0/1456544149 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): get_auth_request con 0x7fae6015f480 auth_method 0 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:10.927 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae65739640 10 monclient(hunting): get_auth_request con 0x7fae60151ff0 auth_method 0 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae65739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae65739640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae65739640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.921+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.922+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.922+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.922+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 1 -- 10.20.193.51:0/1456544149 >> v2:10.20.193.125:3300/0 conn(0x7fae60151ff0 msgr2=0x7fae60152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:36:10.928 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 1 --2- 10.20.193.51:0/1456544149 >> v2:10.20.193.125:3300/0 conn(0x7fae60151ff0 0x7fae60152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 1 -- 10.20.193.51:0/1456544149 >> v1:10.20.193.125:6789/0 conn(0x7fae600c84e0 legacy=0x7fae6015d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 10 monclient: _finish_hunting 0 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 1 monclient: found mon.noname-b 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.923+0000 7fae64f38640 1 -- 10.20.193.51:0/1456544149 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae600c8eb0 con 0x7fae6015f480 2026-02-22T21:36:10.929 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.924+0000 7fae64f38640 1 --2- 10.20.193.51:0/1456544149 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae6015f480 0x7fae6015f860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae50009970 tx=0x7fae5002eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e7ac1a040918b39 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:10.933 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 1 -- 10.20.193.51:0/1456544149 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5003c070 con 0x7fae6015f480 2026-02-22T21:36:10.933 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:10.933 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: dump: 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: _finish_auth 0 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: _check_auth_tickets 2026-02-22T21:36:10.934 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 1 -- 10.20.193.51:0/1456544149 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5002fbc0 con 0x7fae6015f480 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae671c3640 5 monclient: authenticate success, global_id 4258 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae671c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 1 -- 10.20.193.51:0/1456544149 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50037440 con 0x7fae6015f480 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae671c3640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae671c3640 4 set_mon_vals no callback set 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.926+0000 7fae577fe640 10 monclient: dump: 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.935 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 10 monclient: shutdown 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 -- 10.20.193.51:0/1456544149 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae6015f480 msgr2=0x7fae6015f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 --2- 10.20.193.51:0/1456544149 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae6015f480 0x7fae6015f860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae50009970 tx=0x7fae5002eef0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 -- 10.20.193.51:0/1456544149 shutdown_connections 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 --2- 10.20.193.51:0/1456544149 >> v2:10.20.193.125:3300/0 conn(0x7fae60151ff0 0x7fae60152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 --2- 10.20.193.51:0/1456544149 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae6015f480 0x7fae6015f860 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 -- 10.20.193.51:0/1456544149 >> 10.20.193.51:0/1456544149 conn(0x7fae6006c890 msgr2=0x7fae6006cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 -- 10.20.193.51:0/1456544149 shutdown_connections 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 -- 10.20.193.51:0/1456544149 wait complete. 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 10 monclient: build_initial_monmap 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 10 monclient: monmap: 2026-02-22T21:36:10.936 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 librados: starting msgr at 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.927+0000 7fae671c3640 1 librados: starting objecter 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 Processor -- start 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 -- start start 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 librados: setting wanted keys 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 librados: calling monclient init 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient: init 2026-02-22T21:36:10.937 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 0x7fae601f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-b con 0x7fae600c84e0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae60151ff0 0x7fae601f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-c con 0x7fae60151ff0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 0x7fae601f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): picked mon.noname-a con 0x7fae6015f480 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fae600c9750 con 0x7fae6015f480 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fae600c9470 con 0x7fae600c84e0 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fae600c9a30 con 0x7fae60151ff0 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae671c3640 10 monclient(hunting): authenticate will time out at 1046.508603s 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.928+0000 7fae64f38640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 0x7fae601f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 0x7fae601f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae60151ff0 0x7fae601f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 0x7fae601f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:32964/0 (socket says 10.20.193.51:32964) 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 1 -- 10.20.193.51:0/3045048013 learned_addr learned my addr 10.20.193.51:0/3045048013 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): get_auth_request con 0x7fae600c84e0 auth_method 0 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): get_auth_request con 0x7fae6015f480 auth_method 0 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): get_auth_request con 0x7fae60151ff0 auth_method 0 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae64f38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:10.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.929+0000 7fae65739640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 -- 10.20.193.51:0/3045048013 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae60151ff0 msgr2=0x7fae601f66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae60151ff0 0x7fae601f66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 -- 10.20.193.51:0/3045048013 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 msgr2=0x7fae601f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 0x7fae601f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 10 monclient: _finish_hunting 0 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 monclient: found mon.noname-a 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:10.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.931+0000 7fae65739640 1 -- 10.20.193.51:0/3045048013 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae601f5050 con 0x7fae6015f480 2026-02-22T21:36:10.941 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.932+0000 7fae65739640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 0x7fae601f7150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00efc0 tx=0x7fae5c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:10.941 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.932+0000 7fae557fa640 10 client.?.objecter ms_handle_connect 0x7fae6015f480 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.932+0000 7fae557fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c018020 con 0x7fae6015f480 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: dump: 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: _finish_auth 0 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: _check_auth_tickets 2026-02-22T21:36:10.942 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c014480 con 0x7fae6015f480 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c014870 con 0x7fae6015f480 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae56ffd640 4 set_mon_vals no callback set 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 5 monclient: authenticate success, global_id 4263 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 10 monclient: _renew_subs 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae601f5330 con 0x7fae6015f480 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae557fa640 10 monclient: dump: 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:10.943 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 10 monclient: _renew_subs 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae601f58f0 con 0x7fae6015f480 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.934+0000 7fae671c3640 1 librados: init done 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.935+0000 7fae671c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.935+0000 7fae671c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.935+0000 7fae671c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:10.944 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.935+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6015f860 con 0x7fae6015f480 2026-02-22T21:36:10.948 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.946+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7fae5c014a10 con 0x7fae6015f480 2026-02-22T21:36:10.948 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.947+0000 7fae557fa640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 0x7fae3007b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:10.949 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.947+0000 7fae64f38640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 0x7fae3007b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:10.949 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.947+0000 7fae64f38640 10 monclient: get_auth_request con 0x7fae30078dc0 auth_method 0 2026-02-22T21:36:10.949 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.947+0000 7fae64f38640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 0x7fae3007b280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae5002ef30 tx=0x7fae50039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:10.949 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:10.947+0000 7fae557fa640 10 client.4263.objecter ms_handle_connect 0x7fae30078dc0 2026-02-22T21:36:11.054 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7fae5c05ef10 con 0x7fae6015f480 2026-02-22T21:36:11.054 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 10 client.4263.objecter ms_dispatch2 0x7fae60000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:11.054 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 3 client.4263.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:11.054 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 3 client.4263.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:11.055 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 20 client.4263.objecter dump_active .. 0 homeless 2026-02-22T21:36:11.055 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fae5c022030 con 0x7fae6015f480 2026-02-22T21:36:11.055 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:11.055 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.052+0000 7fae557fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:11.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.139+0000 7fae671c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:11.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.139+0000 7fae671c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:11.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.139+0000 7fae671c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:11.141 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.139+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae600c7e80 con 0x7fae6015f480 2026-02-22T21:36:11.166 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.164+0000 7fae557fa640 1 -- 10.20.193.51:0/3045048013 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7fae5c010040 con 0x7fae6015f480 2026-02-22T21:36:11.166 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.164+0000 7fae557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:11.166 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.164+0000 7fae557fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:11.166 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:11.167 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:11.169 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 10 librados: watch_flush enter 2026-02-22T21:36:11.169 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 10 librados: watch_flush exit 2026-02-22T21:36:11.169 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 20 client.4263.objecter shutdown clearing up homeless session... 2026-02-22T21:36:11.169 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 10 client.4263.objecter successfully canceled tick 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 msgr2=0x7fae3007b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 0x7fae3007b280 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae5002ef30 tx=0x7fae50039040 comp rx=0 tx=0).stop 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 10 monclient: shutdown 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 msgr2=0x7fae601f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.166+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 0x7fae601f7150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00efc0 tx=0x7fae5c00c490 comp rx=0 tx=0).stop 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 shutdown_connections 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae30078dc0 0x7fae3007b280 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae6015f480 0x7fae601f7150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae60151ff0 0x7fae601f66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:11.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 --2- 10.20.193.51:0/3045048013 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae600c84e0 0x7fae601f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:11.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 >> 10.20.193.51:0/3045048013 conn(0x7fae6006c890 msgr2=0x7fae601ea400 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:11.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 shutdown_connections 2026-02-22T21:36:11.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 -- 10.20.193.51:0/3045048013 wait complete. 2026-02-22T21:36:11.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:11.167+0000 7fae671c3640 1 librados: shutdown 2026-02-22T21:36:11.178 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:13.272 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:20.164 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.506 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:20.449 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.548 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:20.458 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.597 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:20.489 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.619 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:20.522 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.792 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:20.734 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.801 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:20.693 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.823 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:20.759 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.843 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:20.786 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.874 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:20.814 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.875 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:20.784 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.882 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:20.774 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.908 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:20.800 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.927 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:20.832 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.938 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:20.842 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:13.987 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:20.890 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.044 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:20.984 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.108 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:21.000 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.117 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:21.060 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.119 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:21.061 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.121 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:21.013 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.143 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:21.080 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.152 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:21.044 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.158 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:21.101 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.183 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:21.088 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.198 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:21.107 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.207 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:21.147 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.226 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:21.119 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.251 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:21.155 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.257 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:21.162 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.308 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:21.245 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.319 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:21.222 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.373 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:21.313 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.423 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:21.315 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.426 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:21.318 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.427 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:21.319 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.432 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:21.374 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.441 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:21.384 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.452 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:21.395 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.456 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:21.393 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.488 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:21.397 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.502 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:21.407 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.516 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:21.456 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.525 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:21.429 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.532 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:21.425 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.552 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:21.457 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.625 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:21.528 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.629 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:21.567 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.689 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:21.581 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.693 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:21.585 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.693 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:21.633 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.709 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:21.651 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.730 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:21.667 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.733 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:21.676 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.739 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:21.632 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.740 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:21.683 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.762 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:21.671 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.790 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:21.730 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.806 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:21.699 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.808 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:21.713 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.814 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:21.718 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.825 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:21.730 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.899 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:21.802 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.936 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:21.873 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.955 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:21.847 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.961 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:21.853 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.968 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:21.908 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:14.984 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:21.926 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.004 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:21.941 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.009 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:21.952 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.014 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:21.906 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.019 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:21.963 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.036 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:21.945 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.067 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:22.008 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.081 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:21.973 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.082 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:21.987 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.097 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:22.002 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.098 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:22.001 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.173 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:22.076 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.211 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:22.148 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.218 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:22.110 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.234 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.126 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.244 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:22.184 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.259 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:22.201 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.283 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:22.220 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.288 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:22.180 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.289 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:22.232 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.295 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:22.238 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.314 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:22.223 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.341 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:22.281 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.355 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:22.247 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.359 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:22.264 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.363 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:22.268 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.372 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:22.276 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.451 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:22.354 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.481 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:22.373 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.489 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:22.426 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.508 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.400 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.520 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:22.460 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.533 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:22.475 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.558 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:22.495 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.559 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:22.502 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.562 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:22.454 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.575 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:22.518 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.590 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:22.499 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.618 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:22.559 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.630 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:22.535 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.632 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:22.525 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.635 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:22.540 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.659 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:22.563 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.725 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:22.628 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.745 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:22.637 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.763 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:22.700 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.796 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:22.736 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.807 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:22.749 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.829 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:22.772 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.839 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:22.776 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.853 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:22.745 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.864 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.756 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.867 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:22.810 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.868 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:22.777 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.894 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:22.799 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.898 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:22.839 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.910 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.802 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.917 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:22.809 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.926 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:22.831 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.942 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:22.845 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.946 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.838 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.968 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.860 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.972 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:22.864 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:15.998 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:22.902 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.009 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:22.901 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.047 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:22.984 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.075 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.015 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.085 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.028 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.101 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:23.044 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.123 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.060 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.129 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:23.021 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.158 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.063 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.182 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.123 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.191 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:23.083 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.199 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:23.105 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.245 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:23.189 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.256 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:23.165 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.284 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:23.228 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.284 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.188 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.305 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:23.248 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.318 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:23.222 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.322 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:23.259 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.329 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:23.273 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.334 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:23.277 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.355 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.294 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.370 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:23.262 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.372 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:23.315 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.425 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:23.317 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.431 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:23.323 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.457 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:23.349 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.469 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.412 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.489 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:23.381 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.493 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:23.398 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.495 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:23.387 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.498 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.435 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.517 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.422 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.546 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:23.455 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.553 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.458 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.570 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.510 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.571 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:23.480 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.579 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.484 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.586 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:23.478 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.599 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:23.536 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.621 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:23.525 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.630 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.570 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.639 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:23.543 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.662 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.565 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.682 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:23.585 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.687 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:23.590 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.706 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.609 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.737 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.677 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.743 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:23.686 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.744 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.647 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.773 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.713 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.774 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.716 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.792 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.729 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.799 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.739 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.807 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.750 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.814 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.751 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.823 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.763 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.828 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:23.767 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.828 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:23.721 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.845 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.781 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.848 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.791 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.849 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.754 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.849 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:23.758 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.850 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:23.787 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.854 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:23.796 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.854 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:23.763 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.854 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:23.759 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.871 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:23.776 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.884 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:23.776 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.917 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.857 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.944 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.884 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.949 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:23.889 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:16.995 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:23.932 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.019 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.922 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.024 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:23.927 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.055 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:23.999 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.098 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:24.041 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.134 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:24.027 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.135 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:24.078 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.139 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:24.082 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.165 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:24.057 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.171 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:24.063 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.184 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:24.089 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.201 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:24.093 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.206 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:24.098 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.224 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:24.129 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.259 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:24.164 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.263 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:24.169 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.299 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:24.236 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.336 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:24.273 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.386 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:24.323 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.391 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:24.329 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.450 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:24.342 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:17.456 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:24.348 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:18.786 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 10 monclient: get_monmap_and_config 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 10 monclient: build_initial_monmap 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 10 monclient: monmap: 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:18.847805+0000 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:18.847805+0000 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:18.848 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 1 Processor -- start 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 1 -- start start 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.847+0000 7fcf03a2b640 10 monclient: init 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient: _add_conns ranks=[5,0,4,1,2,3] 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 0x7fcefc074010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.849 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc073c30 addr v2:10.20.193.125:3300/0 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcefc0cc200 addr v1:10.20.193.51:6789/0 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fcefc15c0f0 0x7fcefc15c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc15c0f0 addr v2:10.20.193.68:3300/0 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcefc0cca10 con 0x7fcefc0cc200 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fcefc0cccf0 con 0x7fcefc15c0f0 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fcefc0ccfd0 con 0x7fcefc073c30 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:18.850 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.848+0000 7fcf03a2b640 10 monclient(hunting): authenticate will time out at 1054.428289s 2026-02-22T21:36:18.873 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.870+0000 7fcf017a0640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 0x7fcefc074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.873 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 0x7fcefc074010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:42574/0 (socket says 10.20.193.51:42574) 2026-02-22T21:36:18.873 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 1 -- 10.20.193.51:0/2337875006 learned_addr learned my addr 10.20.193.51:0/2337875006 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:18.873 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): get_auth_request con 0x7fcefc073c30 auth_method 0 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 1 --2- 10.20.193.51:0/2337875006 >> v2:10.20.193.68:3300/0 conn(0x7fcefc15c0f0 0x7fcefc15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): get_auth_request con 0x7fcefc15c0f0 auth_method 0 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:18.874 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:18.875 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:18.875 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.871+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 1 -- 10.20.193.51:0/2337875006 >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 msgr2=0x7fcefc074010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 1 --2- 10.20.193.51:0/2337875006 >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 0x7fcefc074010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 1 -- 10.20.193.51:0/2337875006 >> v1:10.20.193.51:6789/0 conn(0x7fcefc0cc200 legacy=0x7fcefc15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 10 monclient: _finish_hunting 0 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 1 monclient: found mon.noname-b 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:18.876 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.872+0000 7fcf01fa1640 1 -- 10.20.193.51:0/2337875006 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc0cd2b0 con 0x7fcefc15c0f0 2026-02-22T21:36:18.877 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.874+0000 7fcf01fa1640 1 --2- 10.20.193.51:0/2337875006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15c540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0cf380 tx=0x7fcef802f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28437942aa8ec152 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:18.881 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 1 -- 10.20.193.51:0/2337875006 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef803c070 con 0x7fcefc15c0f0 2026-02-22T21:36:18.881 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:18.881 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: dump: 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:18.882 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: _finish_auth 0 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: _check_auth_tickets 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 1 -- 10.20.193.51:0/2337875006 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef802fe30 con 0x7fcefc15c0f0 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 1 -- 10.20.193.51:0/2337875006 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef8037640 con 0x7fcefc15c0f0 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fceebfff640 10 monclient: dump: 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:18.883 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:18.884 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 5 monclient: authenticate success, global_id 4264 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 4 set_mon_vals no callback set 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 10 monclient: shutdown 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 -- 10.20.193.51:0/2337875006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc15c0f0 msgr2=0x7fcefc15c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:18.885 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/2337875006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15c540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0cf380 tx=0x7fcef802f1c0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 -- 10.20.193.51:0/2337875006 shutdown_connections 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/2337875006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15c540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/2337875006 >> v2:10.20.193.125:3300/0 conn(0x7fcefc073c30 0x7fcefc074010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 -- 10.20.193.51:0/2337875006 >> 10.20.193.51:0/2337875006 conn(0x7fcefc0d1880 msgr2=0x7fcefc0d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 -- 10.20.193.51:0/2337875006 shutdown_connections 2026-02-22T21:36:18.886 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.876+0000 7fcf03a2b640 1 -- 10.20.193.51:0/2337875006 wait complete. 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient: build_initial_monmap 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient: monmap: 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:18.887 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.888 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 librados: starting msgr at 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 librados: starting objecter 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 Processor -- start 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 -- start start 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 librados: setting wanted keys 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 librados: calling monclient init 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient: init 2026-02-22T21:36:18.889 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-b con 0x7fcefc073c30 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcefc0cc200 0x7fcefc1f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-a con 0x7fcefc0cc200 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): picked mon.noname-c con 0x7fcefc15c0f0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:18.890 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc15ca80 con 0x7fcefc0cc200 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc0cd870 con 0x7fcefc073c30 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc15cc00 con 0x7fcefc15c0f0 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.877+0000 7fcf03a2b640 10 monclient(hunting): authenticate will time out at 1054.457783s 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcefc0cc200 0x7fcefc1f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:44412/0 (socket says 10.20.193.51:44412) 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.891 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 1 -- 10.20.193.51:0/3223671307 learned_addr learned my addr 10.20.193.51:0/3223671307 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): get_auth_request con 0x7fcefc073c30 auth_method 0 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): get_auth_request con 0x7fcefc0cc200 auth_method 0 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf017a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:18.892 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): get_auth_request con 0x7fcefc15c0f0 auth_method 0 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf00f9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.878+0000 7fcf01fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 -- 10.20.193.51:0/3223671307 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcefc15c0f0 msgr2=0x7fcefc15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.893 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 -- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcefc0cc200 msgr2=0x7fcefc1f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcefc0cc200 0x7fcefc1f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 10 monclient: _finish_hunting 0 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 monclient: found mon.noname-b 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:18.894 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.879+0000 7fcf017a0640 1 -- 10.20.193.51:0/3223671307 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc0cf820 con 0x7fcefc073c30 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.880+0000 7fcf017a0640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e930 tx=0x7fceec00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.880+0000 7fcee9ffb640 10 client.?.objecter ms_handle_connect 0x7fcefc073c30 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.880+0000 7fcee9ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec017070 con 0x7fcefc073c30 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:18.897 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: dump: 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: _finish_auth 0 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:18.898 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 5 monclient: authenticate success, global_id 4267 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _renew_subs 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcefc0cfb00 con 0x7fcefc073c30 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _renew_subs 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcefc0d00c0 con 0x7fcefc073c30 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 1 librados: init done 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceec005a90 con 0x7fcefc073c30 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec021060 con 0x7fcefc073c30 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:18.899 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcee9ffb640 10 monclient: dump: 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fceeb7fe640 4 set_mon_vals no callback set 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:18.900 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.882+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcefc0d12c0 con 0x7fcefc073c30 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.893+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7fceec024060 con 0x7fcefc073c30 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.894+0000 7fcee9ffb640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 0x7fcecc07b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.894+0000 7fcf00f9f640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 0x7fcecc07b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.894+0000 7fcf00f9f640 10 monclient: get_auth_request con 0x7fcecc078d70 auth_method 0 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.894+0000 7fcee9ffb640 10 client.4267.objecter ms_handle_connect 0x7fcecc078d70 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.894+0000 7fcf00f9f640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 0x7fcecc07b230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcefc15ebe0 tx=0x7fcef0007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.895+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7fceec05dd00 con 0x7fcefc073c30 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.895+0000 7fcee9ffb640 10 client.4267.objecter ms_dispatch2 0x7fcefc000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.895+0000 7fcee9ffb640 3 client.4267.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.895+0000 7fcee9ffb640 3 client.4267.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:18.901 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:18.895+0000 7fcee9ffb640 20 client.4267.objecter dump_active .. 0 homeless 2026-02-22T21:36:19.002 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.000+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fceec061cc0 con 0x7fcefc073c30 2026-02-22T21:36:19.002 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.000+0000 7fcee9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:19.002 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.000+0000 7fcee9ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:19.090 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.088+0000 7fcf03a2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:19.090 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.088+0000 7fcf03a2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:19.090 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.088+0000 7fcf03a2b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:36:19.091 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.088+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcefc0742d0 con 0x7fcefc073c30 2026-02-22T21:36:19.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.099+0000 7fcee9ffb640 1 -- 10.20.193.51:0/3223671307 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7fceec065fc0 con 0x7fcefc073c30 2026-02-22T21:36:19.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.099+0000 7fcee9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:19.101 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.099+0000 7fcee9ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:19.102 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:19.102 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:19.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 10 librados: watch_flush enter 2026-02-22T21:36:19.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 10 librados: watch_flush exit 2026-02-22T21:36:19.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 20 client.4267.objecter shutdown clearing up homeless session... 2026-02-22T21:36:19.104 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 10 client.4267.objecter successfully canceled tick 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 msgr2=0x7fcecc07b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 0x7fcecc07b230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcefc15ebe0 tx=0x7fcef0007f30 comp rx=0 tx=0).stop 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 10 monclient: shutdown 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 msgr2=0x7fcefc0d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e930 tx=0x7fceec00ee00 comp rx=0 tx=0).stop 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 shutdown_connections 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcecc078d70 0x7fcecc07b230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcefc15c0f0 0x7fcefc15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcefc0cc200 0x7fcefc1f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:19.105 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 --2- 10.20.193.51:0/3223671307 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcefc073c30 0x7fcefc0d0ba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:19.106 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 >> 10.20.193.51:0/3223671307 conn(0x7fcefc0d1880 msgr2=0x7fcefc1544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:19.106 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 shutdown_connections 2026-02-22T21:36:19.106 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 -- 10.20.193.51:0/3223671307 wait complete. 2026-02-22T21:36:19.106 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:19.102+0000 7fcf03a2b640 1 librados: shutdown 2026-02-22T21:36:19.112 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:26.722 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:26.785 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 10 monclient: get_monmap_and_config 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 10 monclient: build_initial_monmap 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 10 monclient: monmap: 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:26.784896+0000 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:26.784896+0000 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:26.786 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:26.787 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 1 Processor -- start 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 1 -- start start 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.783+0000 7f79a31a0640 10 monclient: init 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient: _add_conns ranks=[4,1,0,3,2,5] 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f799c152650 0x7f799c15d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-b con 0x7f799c152650 addr v2:10.20.193.68:3300/0 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f799c15d6e0 addr v1:10.20.193.68:6789/0 2026-02-22T21:36:26.790 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f799c0703d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c0beb80 con 0x7f799c0703d0 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c0bee60 con 0x7f799c15d6e0 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f799c0bf140 con 0x7f799c152650 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a31a0640 10 monclient(hunting): authenticate will time out at 1062.365301s 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a1716640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f799c0703d0 0x7f799c0c7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:34280/0 (socket says 10.20.193.51:34280) 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a1716640 1 -- 10.20.193.51:0/3961624636 learned_addr learned my addr 10.20.193.51:0/3961624636 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.784+0000 7f79a0f15640 1 --2- 10.20.193.51:0/3961624636 >> v2:10.20.193.68:3300/0 conn(0x7f799c152650 0x7f799c15d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): get_auth_request con 0x7f799c152650 auth_method 0 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:26.791 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.785+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 1 -- 10.20.193.51:0/3961624636 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561402315 0 0) 0x7f799c0beb80 con 0x7f799c0703d0 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 10 monclient(hunting): _negotiate 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 10 monclient(hunting): my global_id is 4269 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.786+0000 7f79937fe640 1 -- 10.20.193.51:0/3961624636 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7988003620 con 0x7f799c0703d0 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2026-02-22T21:36:26.792 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 1 -- 10.20.193.51:0/3961624636 >> v1:10.20.193.68:6789/0 conn(0x7f799c15d6e0 legacy=0x7f799c06fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 1 -- 10.20.193.51:0/3961624636 >> v1:10.20.193.51:6789/0 conn(0x7f799c0703d0 legacy=0x7f799c0c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 10 monclient: _finish_hunting 0 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 1 monclient: found mon.noname-b 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:26.793 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.787+0000 7f79a0f15640 1 -- 10.20.193.51:0/3961624636 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c0bf420 con 0x7f799c152650 2026-02-22T21:36:26.797 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.789+0000 7f79a0f15640 1 --2- 10.20.193.51:0/3961624636 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15d1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098a0 tx=0x7f798c02ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb5a588d6371d55d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:26.797 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 1 -- 10.20.193.51:0/3961624636 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c03c070 con 0x7f799c152650 2026-02-22T21:36:26.798 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:26.798 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:26.798 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: dump: 2026-02-22T21:36:26.798 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:26.798 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:26.799 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: _finish_auth 0 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: _check_auth_tickets 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 1 -- 10.20.193.51:0/3961624636 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c02fc10 con 0x7f799c152650 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 5 monclient: authenticate success, global_id 4270 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 1 -- 10.20.193.51:0/3961624636 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c037440 con 0x7f799c152650 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:26.800 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79937fe640 10 monclient: dump: 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:26.801 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 4 set_mon_vals no callback set 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 10 monclient: shutdown 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 -- 10.20.193.51:0/3961624636 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 msgr2=0x7f799c15d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:26.802 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 --2- 10.20.193.51:0/3961624636 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15d1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098a0 tx=0x7f798c02ef00 comp rx=0 tx=0).stop 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 -- 10.20.193.51:0/3961624636 shutdown_connections 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 --2- 10.20.193.51:0/3961624636 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15d1a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 -- 10.20.193.51:0/3961624636 >> 10.20.193.51:0/3961624636 conn(0x7f799c0cd4f0 msgr2=0x7f799c0cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.791+0000 7f79a31a0640 1 -- 10.20.193.51:0/3961624636 shutdown_connections 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 -- 10.20.193.51:0/3961624636 wait complete. 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 10 monclient: build_initial_monmap 2026-02-22T21:36:26.803 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 10 monclient: monmap: 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:26.804 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 librados: starting msgr at 2026-02-22T21:36:26.805 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 librados: starting objecter 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 Processor -- start 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 -- start start 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 librados: setting wanted keys 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 1 librados: calling monclient init 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.792+0000 7f79a31a0640 10 monclient: init 2026-02-22T21:36:26.806 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:26.807 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:36:26.807 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:26.807 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-a con 0x7f799c0703d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:26.807 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:26.807 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-b con 0x7f799c152650 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f799c15d6e0 0x7f799c14e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): picked mon.noname-c con 0x7f799c15d6e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f799c0be800 con 0x7f799c0703d0 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:26.808 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f799c0bf9e0 con 0x7f799c152650 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f799c152a30 con 0x7f799c15d6e0 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a31a0640 10 monclient(hunting): authenticate will time out at 1062.374189s 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:26.809 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f7993fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:26.810 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:54272/0 (socket says 10.20.193.51:54272) 2026-02-22T21:36:26.810 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 1 -- 10.20.193.51:0/930499007 learned_addr learned my addr 10.20.193.51:0/930499007 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:26.810 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 10 monclient(hunting): get_auth_request con 0x7f799c0703d0 auth_method 0 2026-02-22T21:36:26.810 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:26.810 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a0f15640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.793+0000 7f79a1716640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f799c15d6e0 0x7f799c14e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a0f15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:26.811 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): get_auth_request con 0x7f799c15d6e0 auth_method 0 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:26.812 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.794+0000 7f79a1716640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:26.818 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): get_auth_request con 0x7f799c152650 auth_method 0 2026-02-22T21:36:26.819 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:26.819 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:26.819 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:26.819 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:26.819 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f7993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 -- 10.20.193.51:0/930499007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f799c15d6e0 msgr2=0x7f799c14e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f799c15d6e0 0x7f799c14e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 -- 10.20.193.51:0/930499007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 msgr2=0x7f799c15b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:26.820 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 10 monclient: _finish_hunting 0 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 monclient: found mon.noname-a 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.795+0000 7f79a0f15640 1 -- 10.20.193.51:0/930499007 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c14a0a0 con 0x7f799c0703d0 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.797+0000 7f79a0f15640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f798c009130 tx=0x7f798c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.797+0000 7f79917fa640 10 client.?.objecter ms_handle_connect 0x7f799c0703d0 2026-02-22T21:36:26.821 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.797+0000 7f79917fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c03c040 con 0x7f799c0703d0 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: dump: 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:26.822 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:26.823 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: _finish_auth 0 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: _check_auth_tickets 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 5 monclient: authenticate success, global_id 4272 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c042630 con 0x7f799c0703d0 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:26.824 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c042a20 con 0x7f799c0703d0 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f7992ffd640 4 set_mon_vals no callback set 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79917fa640 10 monclient: dump: 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.825 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 10 monclient: _renew_subs 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:26.826 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c1f79a0 con 0x7f799c0703d0 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 10 monclient: _renew_subs 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c1f3c40 con 0x7f799c0703d0 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.798+0000 7f79a31a0640 1 librados: init done 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.799+0000 7f79a31a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.799+0000 7f79a31a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.799+0000 7f79a31a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:26.827 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.799+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c15d1a0 con 0x7f799c0703d0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.810+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f798c04b450 con 0x7f799c0703d0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.810+0000 7f79917fa640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 0x7f796c07b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.810+0000 7f7993fff640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 0x7f796c07b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.811+0000 7f7993fff640 10 monclient: get_auth_request con 0x7f796c078d70 auth_method 0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.811+0000 7f79917fa640 10 client.4272.objecter ms_handle_connect 0x7f796c078d70 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.811+0000 7f7993fff640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 0x7f796c07b230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f799c0cc8c0 tx=0x7f7984007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.813+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f798c02fb50 con 0x7f799c0703d0 2026-02-22T21:36:26.828 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.813+0000 7f79917fa640 10 client.4272.objecter ms_dispatch2 0x7f799c000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:26.829 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.813+0000 7f79917fa640 3 client.4272.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:26.829 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.813+0000 7f79917fa640 3 client.4272.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:26.829 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.813+0000 7f79917fa640 20 client.4272.objecter dump_active .. 0 homeless 2026-02-22T21:36:26.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.920+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f799c14a0a0 con 0x7f799c0703d0 2026-02-22T21:36:26.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.920+0000 7f79917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:26.923 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:26.920+0000 7f79917fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:27.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.011+0000 7f79a31a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:27.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.011+0000 7f79a31a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:27.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.011+0000 7f79a31a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:27.014 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.011+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f799c15f890 con 0x7f799c0703d0 2026-02-22T21:36:27.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.021+0000 7f79917fa640 1 -- 10.20.193.51:0/930499007 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f798c0868f0 con 0x7f799c0703d0 2026-02-22T21:36:27.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.021+0000 7f79917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:27.024 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.021+0000 7f79917fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:27.024 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:27.025 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:27.027 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 10 librados: watch_flush enter 2026-02-22T21:36:27.027 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 10 librados: watch_flush exit 2026-02-22T21:36:27.027 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 20 client.4272.objecter shutdown clearing up homeless session... 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 10 client.4272.objecter successfully canceled tick 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 msgr2=0x7f796c07b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 0x7f796c07b230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f799c0cc8c0 tx=0x7f7984007ef0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 10 monclient: shutdown 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 msgr2=0x7f799c150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f798c009130 tx=0x7f798c0047c0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 shutdown_connections 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f796c078d70 0x7f796c07b230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f799c15d6e0 0x7f799c14e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.028 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f799c152650 0x7f799c15b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.029 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 --2- 10.20.193.51:0/930499007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f799c0703d0 0x7f799c150ec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:27.029 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 >> 10.20.193.51:0/930499007 conn(0x7f799c0cd4f0 msgr2=0x7f799c14ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:27.029 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 shutdown_connections 2026-02-22T21:36:27.029 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 -- 10.20.193.51:0/930499007 wait complete. 2026-02-22T21:36:27.029 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:27.024+0000 7f79a31a0640 1 librados: shutdown 2026-02-22T21:36:27.035 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:27.359 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:34.250 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:27.950 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:34.842 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:27.987 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:34.930 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.368 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:35.271 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.450 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:35.390 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.452 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:35.361 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.455 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:35.392 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.516 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:35.458 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.545 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:35.450 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.674 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:35.615 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.710 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:35.602 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.721 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:35.624 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.844 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:35.787 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.923 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:35.816 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:28.926 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:35.831 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:29.032 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:35.969 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.056 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:36.948 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.346 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:37.238 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.349 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:37.241 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.623 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:37.515 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.645 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:37.588 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.652 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:37.544 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.655 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:37.547 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.682 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:37.625 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.685 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:37.628 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.906 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:37.809 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.951 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:37.891 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.989 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:37.929 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:30.992 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:37.932 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.084 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:38.020 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.087 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:37.996 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.116 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:38.058 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.118 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:38.027 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.121 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:38.030 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.132 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:38.035 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.138 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:38.081 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.141 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:38.084 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.159 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:38.062 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.162 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:38.065 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.185 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:38.089 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.188 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:38.092 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.225 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:38.130 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.256 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:38.161 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.259 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:38.164 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.302 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:38.194 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.321 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:38.261 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.338 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:38.230 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.341 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:38.233 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.371 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:38.307 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.373 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:38.310 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.499 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:38.441 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.527 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:38.470 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.529 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:38.472 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.559 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:38.451 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.564 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:38.469 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.589 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:38.494 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.592 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:38.497 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.594 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:38.486 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.597 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:38.489 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.599 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:38.539 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.602 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:38.542 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:31.713 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:38.650 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.001 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:38.939 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.004 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:38.942 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.039 INFO:tasks.ceph.osd.15.trial140.stderr:==00:00:00:38.931 45037== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.358 INFO:tasks.ceph.osd.7.trial140.stderr:==00:00:00:39.250 45036== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.526 INFO:tasks.ceph.osd.0.trial051.stderr:==00:00:00:39.469 45070== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.756 INFO:tasks.ceph.osd.5.trial068.stderr:==00:00:00:39.696 44652== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.811 INFO:tasks.ceph.osd.13.trial068.stderr:==00:00:00:39.720 44662== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.838 INFO:tasks.ceph.osd.4.trial051.stderr:==00:00:00:39.780 45072== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:32.884 INFO:tasks.ceph.osd.14.trial125.stderr:==00:00:00:39.787 44671== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.026 INFO:tasks.ceph.osd.10.trial125.stderr:==00:00:00:39.929 44670== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.073 INFO:tasks.ceph.osd.2.trial125.stderr:==00:00:00:39.978 44668== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.107 INFO:tasks.ceph.osd.11.trial140.stderr:==00:00:00:39.999 45035== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.216 INFO:tasks.ceph.osd.9.trial068.stderr:==00:00:00:40.153 44655== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.329 INFO:tasks.ceph.osd.12.trial051.stderr:==00:00:00:40.269 45073== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.354 INFO:tasks.ceph.osd.8.trial051.stderr:==00:00:00:40.297 45071== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.423 INFO:tasks.ceph.osd.3.trial140.stderr:==00:00:00:40.315 45034== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.435 INFO:tasks.ceph.osd.6.trial125.stderr:==00:00:00:40.341 44669== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:33.829 INFO:tasks.ceph.osd.1.trial068.stderr:==00:00:00:40.767 44656== Warning: unimplemented fcntl command: 1038 2026-02-22T21:36:34.645 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:34.707 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.704+0000 7f880e0bd640 10 monclient: get_monmap_and_config 2026-02-22T21:36:34.707 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.704+0000 7f880e0bd640 10 monclient: build_initial_monmap 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.704+0000 7f880e0bd640 10 monclient: monmap: 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:34.706522+0000 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:34.706522+0000 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:34.708 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:34.709 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:34.709 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:34.709 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:34.709 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:34.709 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.705+0000 7f880e0bd640 1 Processor -- start 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.705+0000 7f880e0bd640 1 -- start start 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.705+0000 7f880e0bd640 10 monclient: init 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient: _add_conns ranks=[2,5,4,0,3,1] 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88080c9050 addr v1:10.20.193.125:6789/0 2026-02-22T21:36:34.713 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f88080c5fd0 0x7f88080c9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-c con 0x7f88080c5fd0 addr v2:10.20.193.125:3300/0 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8808152570 0x7f88081529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-b con 0x7f8808152570 addr v2:10.20.193.68:3300/0 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880806dfc0 con 0x7f88080c9050 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f880806e260 con 0x7f8808152570 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:34.714 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f880806e540 con 0x7f88080c5fd0 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f880e0bd640 10 monclient(hunting): authenticate will time out at 1070.287315s 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f8807fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8808152570 0x7f88081529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.706+0000 7f8806ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f88080c5fd0 0x7f88080c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8808152570 0x7f88081529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:50896/0 (socket says 10.20.193.51:50896) 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 1 -- 10.20.193.51:0/2111065406 learned_addr learned my addr 10.20.193.51:0/2111065406 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:34.715 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): get_auth_request con 0x7f8808152570 auth_method 0 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): get_auth_request con 0x7f88080c5fd0 auth_method 0 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:34.716 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.707+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 1 -- 10.20.193.51:0/2111065406 >> v2:10.20.193.125:3300/0 conn(0x7f88080c5fd0 msgr2=0x7f88080c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 1 --2- 10.20.193.51:0/2111065406 >> v2:10.20.193.125:3300/0 conn(0x7f88080c5fd0 0x7f88080c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 1 -- 10.20.193.51:0/2111065406 >> v1:10.20.193.125:6789/0 conn(0x7f88080c9050 legacy=0x7f88080c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 10 monclient: _finish_hunting 0 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 1 monclient: found mon.noname-b 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.708+0000 7f8807fff640 1 -- 10.20.193.51:0/2111065406 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880806e820 con 0x7f8808152570 2026-02-22T21:36:34.717 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.710+0000 7f8807fff640 1 --2- 10.20.193.51:0/2111065406 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8808152570 0x7f88081529e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f880806cdf0 tx=0x7f87fc02f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b870d2330c36510 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 1 -- 10.20.193.51:0/2111065406 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc02fe10 con 0x7f8808152570 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: dump: 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:34.723 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:34.724 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: _finish_auth 0 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: _check_auth_tickets 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 1 -- 10.20.193.51:0/2111065406 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc0373f0 con 0x7f8808152570 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 5 monclient: authenticate success, global_id 4279 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:36:34.725 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 1 -- 10.20.193.51:0/2111065406 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0377c0 con 0x7f8808152570 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 4 set_mon_vals no callback set 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f88067fc640 10 monclient: dump: 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:34.726 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 10 monclient: shutdown 2026-02-22T21:36:34.727 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 -- 10.20.193.51:0/2111065406 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8808152570 msgr2=0x7f88081529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 --2- 10.20.193.51:0/2111065406 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8808152570 0x7f88081529e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f880806cdf0 tx=0x7f87fc02f310 comp rx=0 tx=0).stop 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 -- 10.20.193.51:0/2111065406 shutdown_connections 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 --2- 10.20.193.51:0/2111065406 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8808152570 0x7f88081529e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 --2- 10.20.193.51:0/2111065406 >> v2:10.20.193.125:3300/0 conn(0x7f88080c5fd0 0x7f88080c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 -- 10.20.193.51:0/2111065406 >> 10.20.193.51:0/2111065406 conn(0x7f88080cd4f0 msgr2=0x7f88080cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 -- 10.20.193.51:0/2111065406 shutdown_connections 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 1 -- 10.20.193.51:0/2111065406 wait complete. 2026-02-22T21:36:34.728 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 10 monclient: build_initial_monmap 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.712+0000 7f880e0bd640 10 monclient: monmap: 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 librados: starting msgr at 2026-02-22T21:36:34.729 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 librados: starting objecter 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 Processor -- start 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 -- start start 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 librados: setting wanted keys 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 librados: calling monclient init 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient: init 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88080c5fd0 0x7f880814bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-a con 0x7f88080c5fd0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 0x7f88080be800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-b con 0x7f88080c9050 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 0x7f88080bed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): picked mon.noname-c con 0x7f8808152570 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:34.730 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f880815ccf0 con 0x7f88080c5fd0 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f880806ede0 con 0x7f88080c9050 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8808152f20 con 0x7f8808152570 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.713+0000 7f880e0bd640 10 monclient(hunting): authenticate will time out at 1070.294601s 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 0x7f88080bed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 0x7f88080be800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88080c5fd0 0x7f880814bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 0x7f88080be800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:50910/0 (socket says 10.20.193.51:50910) 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 1 -- 10.20.193.51:0/3200995746 learned_addr learned my addr 10.20.193.51:0/3200995746 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:34.731 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): get_auth_request con 0x7f88080c9050 auth_method 0 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): get_auth_request con 0x7f8808152570 auth_method 0 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): get_auth_request con 0x7f88080c5fd0 auth_method 0 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:34.732 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f8806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:34.733 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.714+0000 7f88077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 -- 10.20.193.51:0/3200995746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 msgr2=0x7f88080be800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 0x7f88080be800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 -- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88080c5fd0 msgr2=0x7f880814bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88080c5fd0 0x7f880814bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 10 monclient: _finish_hunting 0 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 monclient: found mon.noname-c 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.716+0000 7f8807fff640 1 -- 10.20.193.51:0/3200995746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8808150bd0 con 0x7f8808152570 2026-02-22T21:36:34.734 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.717+0000 7f8807fff640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 0x7f88080bed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f87fc035000 tx=0x7f87fc004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.717+0000 7f87e3fff640 10 client.?.objecter ms_handle_connect 0x7f8808152570 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.717+0000 7f87e3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc046020 con 0x7f8808152570 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: dump: 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:34.735 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: _finish_auth 0 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc037e30 con 0x7f8808152570 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc009510 con 0x7f8808152570 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f87e3fff640 10 monclient: dump: 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:34.736 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f8805ffb640 4 set_mon_vals no callback set 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 5 monclient: authenticate success, global_id 4289 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 10 monclient: _renew_subs 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8808150eb0 con 0x7f8808152570 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:34.737 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 10 monclient: _renew_subs 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8808151470 con 0x7f8808152570 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.719+0000 7f880e0bd640 1 librados: init done 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.720+0000 7f880e0bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.720+0000 7f880e0bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.720+0000 7f880e0bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:34.738 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.720+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88080c9430 con 0x7f8808152570 2026-02-22T21:36:34.739 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.730+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f87fc007bf0 con 0x7f8808152570 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.731+0000 7f87e3fff640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 0x7f87d807b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.731+0000 7f88077fe640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 0x7f87d807b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.731+0000 7f88077fe640 10 monclient: get_auth_request con 0x7f87d8078d70 auth_method 0 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.731+0000 7f87e3fff640 10 client.4289.objecter ms_handle_connect 0x7f87d8078d70 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.731+0000 7f88077fe640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 0x7f87d807b230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87f4002770 tx=0x7f87f4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.733+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f87fc083bd0 con 0x7f8808152570 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.733+0000 7f87e3fff640 10 client.4289.objecter ms_dispatch2 0x7f8808000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.733+0000 7f87e3fff640 3 client.4289.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.733+0000 7f87e3fff640 3 client.4289.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:34.740 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.733+0000 7f87e3fff640 20 client.4289.objecter dump_active .. 0 homeless 2026-02-22T21:36:34.836 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.834+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f87fc050030 con 0x7f8808152570 2026-02-22T21:36:34.837 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.834+0000 7f87e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:34.837 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.834+0000 7f87e3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:34.924 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.921+0000 7f880e0bd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:34.924 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.921+0000 7f880e0bd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:34.924 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.921+0000 7f880e0bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:34.925 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.921+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88080ca510 con 0x7f8808152570 2026-02-22T21:36:34.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.933+0000 7f87e3fff640 1 -- 10.20.193.51:0/3200995746 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f87fc087b90 con 0x7f8808152570 2026-02-22T21:36:34.935 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.933+0000 7f87e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:34.936 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.933+0000 7f87e3fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:34.936 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:34.936 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:34.938 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 10 librados: watch_flush enter 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 10 librados: watch_flush exit 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 20 client.4289.objecter shutdown clearing up homeless session... 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 10 client.4289.objecter successfully canceled tick 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 msgr2=0x7f87d807b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 0x7f87d807b230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f87f4002770 tx=0x7f87f4007ef0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 10 monclient: shutdown 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 msgr2=0x7f88080bed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:34.939 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 0x7f88080bed40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f87fc035000 tx=0x7f87fc004b40 comp rx=0 tx=0).stop 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 shutdown_connections 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f87d8078d70 0x7f87d807b230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8808152570 0x7f88080bed40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88080c9050 0x7f88080be800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 --2- 10.20.193.51:0/3200995746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88080c5fd0 0x7f880814bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 >> 10.20.193.51:0/3200995746 conn(0x7f88080cd4f0 msgr2=0x7f88080cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 shutdown_connections 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 -- 10.20.193.51:0/3200995746 wait complete. 2026-02-22T21:36:34.940 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:34.935+0000 7f880e0bd640 1 librados: shutdown 2026-02-22T21:36:34.946 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:40.839 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:11.992 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.872 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:12.025 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.890 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:12.042 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.892 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:12.008 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.920 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:12.071 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.920 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:12.036 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:40.989 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:12.143 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:41.037 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:12.189 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:41.038 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:12.154 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:36:42.555 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:42.618 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.616+0000 7f0ea504c640 10 monclient: get_monmap_and_config 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.616+0000 7f0ea504c640 10 monclient: build_initial_monmap 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.616+0000 7f0ea504c640 10 monclient: monmap: 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:42.617323+0000 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:42.617323+0000 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.619 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 Processor -- start 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 -- start start 2026-02-22T21:36:42.620 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient: init 2026-02-22T21:36:42.629 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient: _add_conns ranks=[0,1,5,2,3,4] 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ea0152650 addr v1:10.20.193.51:6789/0 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f0ea006fcc0 addr v1:10.20.193.68:6789/0 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f0ea0164620 0x7f0ea0166a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-c con 0x7f0ea0164620 addr v2:10.20.193.125:3300/0 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea00beb80 con 0x7f0ea0152650 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea00bee60 con 0x7f0ea006fcc0 2026-02-22T21:36:42.630 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea00bf140 con 0x7f0ea0164620 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.617+0000 7f0ea504c640 10 monclient(hunting): authenticate will time out at 1078.197633s 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9ed76640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f0ea0152650 0x7f0ea015d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:35218/0 (socket says 10.20.193.51:35218) 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9ed76640 1 -- 10.20.193.51:0/2534260884 learned_addr learned my addr 10.20.193.51:0/2534260884 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 1 --2- 10.20.193.51:0/2534260884 >> v2:10.20.193.125:3300/0 conn(0x7f0ea0164620 0x7f0ea0166a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): get_auth_request con 0x7f0ea0164620 auth_method 0 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.631 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.618+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151140128 0 0) 0x7f0ea00beb80 con 0x7f0ea0152650 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _negotiate 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): my global_id is 4278 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e8c003620 con 0x7f0ea0152650 2026-02-22T21:36:42.632 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333752250 0 0) 0x7f0ea00bee60 con 0x7f0ea006fcc0 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _negotiate 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 10 monclient(hunting): my global_id is 4285 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.619+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ea00beb80 con 0x7f0ea006fcc0 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 1 -- 10.20.193.51:0/2534260884 >> v1:10.20.193.68:6789/0 conn(0x7f0ea006fcc0 legacy=0x7f0ea0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 1 -- 10.20.193.51:0/2534260884 >> v1:10.20.193.51:6789/0 conn(0x7f0ea0152650 legacy=0x7f0ea015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 10 monclient: _finish_hunting 0 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 1 monclient: found mon.noname-c 2026-02-22T21:36:42.633 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.620+0000 7f0e9f577640 1 -- 10.20.193.51:0/2534260884 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea00bf420 con 0x7f0ea0164620 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.621+0000 7f0e9f577640 1 --2- 10.20.193.51:0/2534260884 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea0164620 0x7f0ea0166a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009d90 tx=0x7f0e9002f2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645803308b146cb1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9003c070 con 0x7f0ea0164620 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 10 monclient: dump: 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.634 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 10 monclient: _finish_auth 0 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 10 monclient: _check_auth_tickets 2026-02-22T21:36:42.635 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0e9dd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0ea504c640 5 monclient: authenticate success, global_id 4292 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.622+0000 7f0ea504c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e90037440 con 0x7f0ea0164620 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 1 -- 10.20.193.51:0/2534260884 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e900377f0 con 0x7f0ea0164620 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0e9dd74640 10 monclient: dump: 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.636 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 4 set_mon_vals no callback set 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 10 monclient: shutdown 2026-02-22T21:36:42.637 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 -- 10.20.193.51:0/2534260884 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea0164620 msgr2=0x7f0ea0166a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 --2- 10.20.193.51:0/2534260884 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea0164620 0x7f0ea0166a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009d90 tx=0x7f0e9002f2c0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 -- 10.20.193.51:0/2534260884 shutdown_connections 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 --2- 10.20.193.51:0/2534260884 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea0164620 0x7f0ea0166a10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 -- 10.20.193.51:0/2534260884 >> 10.20.193.51:0/2534260884 conn(0x7f0ea00cd4f0 msgr2=0x7f0ea00cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 -- 10.20.193.51:0/2534260884 shutdown_connections 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.623+0000 7f0ea504c640 1 -- 10.20.193.51:0/2534260884 wait complete. 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient: build_initial_monmap 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient: monmap: 2026-02-22T21:36:42.638 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.639 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 librados: starting msgr at 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 librados: starting objecter 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 Processor -- start 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 -- start start 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 librados: setting wanted keys 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 librados: calling monclient init 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient: init 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-c con 0x7f0ea006fcc0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:42.640 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0ea0152650 0x7f0ea0151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-b con 0x7f0ea0152650 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0ea0164620 0x7f0ea00cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): picked mon.noname-a con 0x7f0ea0164620 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea00be800 con 0x7f0ea0164620 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea00bf9e0 con 0x7f0ea0152650 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0152a30 con 0x7f0ea006fcc0 2026-02-22T21:36:42.641 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.624+0000 7f0ea504c640 10 monclient(hunting): authenticate will time out at 1078.204818s 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0ea0164620 0x7f0ea00cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:51740/0 (socket says 10.20.193.51:51740) 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 1 -- 10.20.193.51:0/531411567 learned_addr learned my addr 10.20.193.51:0/531411567 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): get_auth_request con 0x7f0ea006fcc0 auth_method 0 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0ea0152650 0x7f0ea0151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:42.642 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): get_auth_request con 0x7f0ea0164620 auth_method 0 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): get_auth_request con 0x7f0ea0152650 auth_method 0 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:42.643 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9f577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.625+0000 7f0e9e575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 -- 10.20.193.51:0/531411567 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0ea0152650 msgr2=0x7f0ea0151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0ea0152650 0x7f0ea0151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 -- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0ea0164620 msgr2=0x7f0ea00cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.644 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0ea0164620 0x7f0ea00cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.645 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:36:42.645 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 monclient: found mon.noname-c 2026-02-22T21:36:42.645 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:42.645 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.645 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.627+0000 7f0e9ed76640 1 -- 10.20.193.51:0/531411567 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea00cbb30 con 0x7f0ea006fcc0 2026-02-22T21:36:42.647 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.628+0000 7f0e9ed76640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c760 tx=0x7f0e9400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.628+0000 7f0e7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0ea006fcc0 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.628+0000 7f0e7f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94017070 con 0x7f0ea006fcc0 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 10 monclient: dump: 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.648 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0e7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 5 monclient: authenticate success, global_id 4295 2026-02-22T21:36:42.649 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 10 monclient: _renew_subs 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea00cbe10 con 0x7f0ea006fcc0 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 10 monclient: _renew_subs 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea0072e00 con 0x7f0ea006fcc0 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.629+0000 7f0ea504c640 1 librados: init done 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0ea504c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0ea504c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:42.650 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0ea504c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea0070130 con 0x7f0ea006fcc0 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e94005bb0 con 0x7f0ea006fcc0 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94021060 con 0x7f0ea006fcc0 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e7f7fe640 10 monclient: dump: 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.651 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:42.652 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.630+0000 7f0e9d573640 4 set_mon_vals no callback set 2026-02-22T21:36:42.742 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.740+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f0e94024080 con 0x7f0ea006fcc0 2026-02-22T21:36:42.743 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.741+0000 7f0e7f7fe640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 0x7f0e6c07b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:42.743 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.741+0000 7f0e9e575640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 0x7f0e6c07b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:42.743 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.741+0000 7f0e9e575640 10 monclient: get_auth_request con 0x7f0e6c078da0 auth_method 0 2026-02-22T21:36:42.744 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.741+0000 7f0e7f7fe640 10 client.4295.objecter ms_handle_connect 0x7f0e6c078da0 2026-02-22T21:36:42.744 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.741+0000 7f0e9e575640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 0x7f0e6c07b260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0e88002410 tx=0x7f0e88003d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:42.745 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.743+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f0e9405d4f0 con 0x7f0ea006fcc0 2026-02-22T21:36:42.745 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.743+0000 7f0e7f7fe640 10 client.4295.objecter ms_dispatch2 0x7f0ea0000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:42.745 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.743+0000 7f0e7f7fe640 3 client.4295.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:42.745 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.743+0000 7f0e7f7fe640 3 client.4295.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:42.746 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.743+0000 7f0e7f7fe640 20 client.4295.objecter dump_active .. 0 homeless 2026-02-22T21:36:42.752 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.751+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0e940614b0 con 0x7f0ea006fcc0 2026-02-22T21:36:42.752 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.751+0000 7f0e7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:42.752 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.751+0000 7f0e7f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:42.843 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.841+0000 7f0ea504c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:42.843 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.841+0000 7f0ea504c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:42.843 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.841+0000 7f0ea504c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:36:42.843 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.841+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ea01596d0 con 0x7f0ea006fcc0 2026-02-22T21:36:42.856 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.854+0000 7f0e7f7fe640 1 -- 10.20.193.51:0/531411567 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f0e940657b0 con 0x7f0ea006fcc0 2026-02-22T21:36:42.856 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.854+0000 7f0e7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:42.856 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.854+0000 7f0e7f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:42.857 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:42.857 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 10 librados: watch_flush enter 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 10 librados: watch_flush exit 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 20 client.4295.objecter shutdown clearing up homeless session... 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 10 client.4295.objecter successfully canceled tick 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 msgr2=0x7f0e6c07b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 0x7f0e6c07b260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0e88002410 tx=0x7f0e88003d50 comp rx=0 tx=0).stop 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 10 monclient: shutdown 2026-02-22T21:36:42.860 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 msgr2=0x7f0ea00cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c760 tx=0x7f0e9400cc30 comp rx=0 tx=0).stop 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 shutdown_connections 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e6c078da0 0x7f0e6c07b260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0ea0164620 0x7f0ea00cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0ea0152650 0x7f0ea0151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 --2- 10.20.193.51:0/531411567 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0ea006fcc0 0x7f0ea00cca00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 >> 10.20.193.51:0/531411567 conn(0x7f0ea00cd4f0 msgr2=0x7f0ea015e500 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 shutdown_connections 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 -- 10.20.193.51:0/531411567 wait complete. 2026-02-22T21:36:42.861 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:42.857+0000 7f0ea504c640 1 librados: shutdown 2026-02-22T21:36:42.867 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:50.478 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:50.536 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 10 monclient: get_monmap_and_config 2026-02-22T21:36:50.536 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 10 monclient: build_initial_monmap 2026-02-22T21:36:50.536 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 10 monclient: monmap: 2026-02-22T21:36:50.536 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:50.536 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:50.535834+0000 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:50.535834+0000 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:50.537 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 1 Processor -- start 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 1 -- start start 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.535+0000 7f51516cd640 10 monclient: init 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient: _add_conns ranks=[1,0,3,4,5,2] 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f514c0c9050 addr v1:10.20.193.68:6789/0 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f514c0c9430 addr v1:10.20.193.51:6789/0 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f514c0c63d0 0x7f514c14eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-a con 0x7f514c0c63d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514c06e340 con 0x7f514c0c9430 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514c06e620 con 0x7f514c0c9050 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f514c06e110 con 0x7f514c0c63d0 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f51516cd640 10 monclient(hunting): authenticate will time out at 1086.116281s 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f514b7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f514c0c63d0 0x7f514c14eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f514affd640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f514c0c9050 0x7f514c06da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:45664/0 (socket says 10.20.193.51:45664) 2026-02-22T21:36:50.538 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.536+0000 7f514affd640 1 -- 10.20.193.51:0/949855133 learned_addr learned my addr 10.20.193.51:0/949855133 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): get_auth_request con 0x7f514c0c63d0 auth_method 0 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.537+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 1 -- 10.20.193.51:0/949855133 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199384527 0 0) 0x7f514c06e620 con 0x7f514c0c9050 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 10 monclient(hunting): my global_id is 4291 2026-02-22T21:36:50.539 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.538+0000 7f5149ffb640 1 -- 10.20.193.51:0/949855133 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51380058b0 con 0x7f514c0c9050 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 1 -- 10.20.193.51:0/949855133 >> v1:10.20.193.68:6789/0 conn(0x7f514c0c9050 legacy=0x7f514c06da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 1 -- 10.20.193.51:0/949855133 >> v1:10.20.193.51:6789/0 conn(0x7f514c0c9430 legacy=0x7f514c0c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.541 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.540+0000 7f514b7fe640 1 -- 10.20.193.51:0/949855133 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c06e900 con 0x7f514c0c63d0 2026-02-22T21:36:50.543 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.541+0000 7f514b7fe640 1 --2- 10.20.193.51:0/949855133 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c63d0 0x7f514c14eef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f513c009d90 tx=0x7f513c02f2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b86560712dbc717 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:50.545 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 1 -- 10.20.193.51:0/949855133 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c03c070 con 0x7f514c0c63d0 2026-02-22T21:36:50.545 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:50.545 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:50.545 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: dump: 2026-02-22T21:36:50.545 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: _finish_auth 0 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 1 -- 10.20.193.51:0/949855133 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513c037440 con 0x7f514c0c63d0 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 1 -- 10.20.193.51:0/949855133 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c037810 con 0x7f514c0c63d0 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f5149ffb640 10 monclient: dump: 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:50.546 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f51516cd640 5 monclient: authenticate success, global_id 4284 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f51516cd640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.543+0000 7f51516cd640 4 set_mon_vals no callback set 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 10 monclient: shutdown 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- 10.20.193.51:0/949855133 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c63d0 msgr2=0x7f514c14eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 --2- 10.20.193.51:0/949855133 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c63d0 0x7f514c14eef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f513c009d90 tx=0x7f513c02f2c0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- 10.20.193.51:0/949855133 shutdown_connections 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 --2- 10.20.193.51:0/949855133 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c63d0 0x7f514c14eef0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- 10.20.193.51:0/949855133 >> 10.20.193.51:0/949855133 conn(0x7f514c0cd4f0 msgr2=0x7f514c0cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- 10.20.193.51:0/949855133 shutdown_connections 2026-02-22T21:36:50.547 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- 10.20.193.51:0/949855133 wait complete. 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 10 monclient: build_initial_monmap 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 10 monclient: monmap: 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 librados: starting msgr at 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 librados: starting objecter 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 Processor -- start 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 -- start start 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 librados: setting wanted keys 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 1 librados: calling monclient init 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.544+0000 7f51516cd640 10 monclient: init 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:50.549 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f514c0c63d0 0x7f514c148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-b con 0x7f514c0c63d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f514c0c9050 0x7f514c1e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-c con 0x7f514c0c9050 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): picked mon.noname-a con 0x7f514c0c9430 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f514c06f1a0 con 0x7f514c0c9430 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f514c06eec0 con 0x7f514c0c63d0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f514c14f430 con 0x7f514c0c9050 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f51516cd640 10 monclient(hunting): authenticate will time out at 1086.125303s 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:41070/0 (socket says 10.20.193.51:41070) 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 1 -- 10.20.193.51:0/2787134188 learned_addr learned my addr 10.20.193.51:0/2787134188 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 10 monclient(hunting): get_auth_request con 0x7f514c0c9430 auth_method 0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514affd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f514c0c63d0 0x7f514c148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.545+0000 7f514a7fc640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f514c0c9050 0x7f514c1e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): get_auth_request con 0x7f514c0c63d0 auth_method 0 2026-02-22T21:36:50.550 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514a7fc640 10 monclient(hunting): get_auth_request con 0x7f514c0c9050 auth_method 0 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514a7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514a7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.546+0000 7f514affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 -- 10.20.193.51:0/2787134188 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f514c0c9050 msgr2=0x7f514c1e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f514c0c9050 0x7f514c1e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 -- 10.20.193.51:0/2787134188 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f514c0c63d0 msgr2=0x7f514c148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f514c0c63d0 0x7f514c148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.551 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.547+0000 7f514b7fe640 1 -- 10.20.193.51:0/2787134188 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c1f47c0 con 0x7f514c0c9430 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.548+0000 7f514b7fe640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f513c03a040 tx=0x7f513c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.548+0000 7f512b7fe640 10 client.?.objecter ms_handle_connect 0x7f514c0c9430 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.548+0000 7f512b7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c03c050 con 0x7f514c0c9430 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:50.552 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: dump: 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513c048070 con 0x7f514c0c9430 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c041a30 con 0x7f514c0c9430 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f512b7fe640 10 monclient: dump: 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:50.553 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51497fa640 4 set_mon_vals no callback set 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 5 monclient: authenticate success, global_id 4290 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 monclient: _renew_subs 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c1f4aa0 con 0x7f514c0c9430 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 monclient: _renew_subs 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c0beb80 con 0x7f514c0c9430 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 1 librados: init done 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.550+0000 7f51516cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.551+0000 7f51516cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.551+0000 7f51516cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.551+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c14f810 con 0x7f514c0c9430 2026-02-22T21:36:50.662 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.661+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f513c041bd0 con 0x7f514c0c9430 2026-02-22T21:36:50.663 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.662+0000 7f512b7fe640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 0x7f511807b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:50.663 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.662+0000 7f514affd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 0x7f511807b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:50.663 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.662+0000 7f514affd640 10 monclient: get_auth_request con 0x7f5118078dc0 auth_method 0 2026-02-22T21:36:50.663 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.662+0000 7f514affd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 0x7f511807b280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5140005980 tx=0x7f5140007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:50.663 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.662+0000 7f512b7fe640 10 client.4290.objecter ms_handle_connect 0x7f5118078dc0 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f513c084390 con 0x7f514c0c9430 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 10 client.4290.objecter ms_dispatch2 0x7f514c000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 3 client.4290.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 3 client.4290.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 20 client.4290.objecter dump_active .. 0 homeless 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f513c088a50 con 0x7f514c0c9430 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:50.672 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.671+0000 7f512b7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:50.761 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.759+0000 7f51516cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:50.761 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.759+0000 7f51516cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:50.761 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.759+0000 7f51516cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:50.761 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.759+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f514c0c85d0 con 0x7f514c0c9430 2026-02-22T21:36:50.772 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.770+0000 7f512b7fe640 1 -- 10.20.193.51:0/2787134188 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f513c088350 con 0x7f514c0c9430 2026-02-22T21:36:50.772 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.770+0000 7f512b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:50.772 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.770+0000 7f512b7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:50.772 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:50.773 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:50.775 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 10 librados: watch_flush enter 2026-02-22T21:36:50.775 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 10 librados: watch_flush exit 2026-02-22T21:36:50.775 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 20 client.4290.objecter shutdown clearing up homeless session... 2026-02-22T21:36:50.775 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 10 client.4290.objecter successfully canceled tick 2026-02-22T21:36:50.775 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 msgr2=0x7f511807b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 0x7f511807b280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5140005980 tx=0x7f5140007dd0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 10 monclient: shutdown 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 msgr2=0x7f514c15b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f513c03a040 tx=0x7f513c004b40 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 shutdown_connections 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5118078dc0 0x7f511807b280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f514c0c9430 0x7f514c15b240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f514c0c9050 0x7f514c1e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 --2- 10.20.193.51:0/2787134188 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f514c0c63d0 0x7f514c148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 >> 10.20.193.51:0/2787134188 conn(0x7f514c0cd4f0 msgr2=0x7f514c14c980 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 shutdown_connections 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 -- 10.20.193.51:0/2787134188 wait complete. 2026-02-22T21:36:50.776 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:50.773+0000 7f51516cd640 1 librados: shutdown 2026-02-22T21:36:50.783 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:36:58.393 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: get_monmap_and_config 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: build_initial_monmap 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: monmap: 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:36:58.454728+0000 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:36:58.454728+0000 2026-02-22T21:36:58.456 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 1 Processor -- start 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 1 -- start start 2026-02-22T21:36:58.457 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: init 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient: _add_conns ranks=[0,3,1,4,2,5] 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.454+0000 7f633b908640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6334152650 addr v1:10.20.193.51:6789/0 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f633406fcc0 0x7f6334070130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 10 monclient(hunting): picked mon.noname-a con 0x7f633406fcc0 addr v2:10.20.193.51:3300/0 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f63340c7c90 addr v1:10.20.193.68:6789/0 2026-02-22T21:36:58.460 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6334070670 con 0x7f6334152650 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63340beb80 con 0x7f63340c7c90 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f63340bee60 con 0x7f633406fcc0 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f633b908640 10 monclient(hunting): authenticate will time out at 1094.035040s 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6338e7c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f633406fcc0 0x7f6334070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6339e7e640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f63340c7c90 0x7f63340c8100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:55768/0 (socket says 10.20.193.51:55768) 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6339e7e640 1 -- 10.20.193.51:0/3233057650 learned_addr learned my addr 10.20.193.51:0/3233057650 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6338e7c640 10 monclient(hunting): get_auth_request con 0x7f633406fcc0 auth_method 0 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6338e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:58.461 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6338e7c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.455+0000 7f6338e7c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.456+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.456+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.456+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 1 -- 10.20.193.51:0/3233057650 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313441628 0 0) 0x7f63340beb80 con 0x7f63340c7c90 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 10 monclient(hunting): _negotiate 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 10 monclient(hunting): my global_id is 4297 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6323fff640 1 -- 10.20.193.51:0/3233057650 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f631c003620 con 0x7f63340c7c90 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2026-02-22T21:36:58.462 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 1 -- 10.20.193.51:0/3233057650 >> v1:10.20.193.68:6789/0 conn(0x7f63340c7c90 legacy=0x7f63340c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 1 -- 10.20.193.51:0/3233057650 >> v1:10.20.193.51:6789/0 conn(0x7f6334152650 legacy=0x7f633415d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 10 monclient: _finish_hunting 0 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 1 monclient: found mon.noname-a 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.463 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.457+0000 7f6338e7c640 1 -- 10.20.193.51:0/3233057650 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63340bf140 con 0x7f633406fcc0 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.459+0000 7f6338e7c640 1 --2- 10.20.193.51:0/3233057650 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f633406fcc0 0x7f6334070130 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63280098a0 tx=0x7f632802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4728358a255043 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 1 -- 10.20.193.51:0/3233057650 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632803c070 con 0x7f633406fcc0 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: dump: 2026-02-22T21:36:58.468 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: _finish_auth 0 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: _check_auth_tickets 2026-02-22T21:36:58.469 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 1 -- 10.20.193.51:0/3233057650 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632802fc10 con 0x7f633406fcc0 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 5 monclient: authenticate success, global_id 4293 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 10 monclient: get_monmap_and_config success 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 1 -- 10.20.193.51:0/3233057650 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328037440 con 0x7f633406fcc0 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 4 set_mon_vals no callback set 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f6323fff640 10 monclient: dump: 2026-02-22T21:36:58.470 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 10 monclient: shutdown 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:58.471 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 -- 10.20.193.51:0/3233057650 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f633406fcc0 msgr2=0x7f6334070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 --2- 10.20.193.51:0/3233057650 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f633406fcc0 0x7f6334070130 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63280098a0 tx=0x7f632802ef00 comp rx=0 tx=0).stop 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 -- 10.20.193.51:0/3233057650 shutdown_connections 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 --2- 10.20.193.51:0/3233057650 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f633406fcc0 0x7f6334070130 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 -- 10.20.193.51:0/3233057650 >> 10.20.193.51:0/3233057650 conn(0x7f63340cd4f0 msgr2=0x7f63340cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 -- 10.20.193.51:0/3233057650 shutdown_connections 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.462+0000 7f633b908640 1 -- 10.20.193.51:0/3233057650 wait complete. 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient: build_initial_monmap 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient: monmap: 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:36:58.472 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 librados: starting msgr at 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 librados: starting objecter 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 Processor -- start 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 -- start start 2026-02-22T21:36:58.473 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 librados: setting wanted keys 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 librados: calling monclient init 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient: init 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient: _reopen_session rank -1 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f633406fcc0 0x7f6334151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): picked mon.noname-b con 0x7f633406fcc0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): picked mon.noname-a con 0x7f63340c7c90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6334152650 0x7f63341f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): picked mon.noname-c con 0x7f6334152650 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:58.474 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f63340bf9e0 con 0x7f63340c7c90 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f63340bf700 con 0x7f633406fcc0 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): start opening mon connection 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f63340be800 con 0x7f6334152650 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): _renew_subs 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.463+0000 7f633b908640 10 monclient(hunting): authenticate will time out at 1094.043964s 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50684/0 (socket says 10.20.193.51:50684) 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 1 -- 10.20.193.51:0/2129677627 learned_addr learned my addr 10.20.193.51:0/2129677627 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): get_auth_request con 0x7f63340c7c90 auth_method 0 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:58.475 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6334152650 0x7f63341f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f633967d640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f633406fcc0 0x7f6334151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6338e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): get_auth_request con 0x7f6334152650 auth_method 0 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f633967d640 10 monclient(hunting): get_auth_request con 0x7f633406fcc0 auth_method 0 2026-02-22T21:36:58.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f633967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f633967d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f633967d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.464+0000 7f6339e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.465+0000 7f633967d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.465+0000 7f633967d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.465+0000 7f633967d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 -- 10.20.193.51:0/2129677627 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6334152650 msgr2=0x7f63341f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6334152650 0x7f63341f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 -- 10.20.193.51:0/2129677627 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f633406fcc0 msgr2=0x7f6334151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f633406fcc0 0x7f6334151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 10 monclient: _finish_hunting 0 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 monclient: found mon.noname-a 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.478 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.466+0000 7f6338e7c640 1 -- 10.20.193.51:0/2129677627 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633415a6d0 con 0x7f63340c7c90 2026-02-22T21:36:58.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.467+0000 7f6338e7c640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f632802f430 tx=0x7f63280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.467+0000 7f6321ffb640 10 client.?.objecter ms_handle_connect 0x7f63340c7c90 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.467+0000 7f6321ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632803c040 con 0x7f63340c7c90 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 10 monclient: dump: 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.481 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 10 monclient: _finish_auth 0 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6328042630 con 0x7f63340c7c90 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f633b908640 5 monclient: authenticate success, global_id 4299 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.468+0000 7f633b908640 10 monclient: _renew_subs 2026-02-22T21:36:58.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633415a9b0 con 0x7f63340c7c90 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f6321ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328042a20 con 0x7f63340c7c90 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f6321ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f6321ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f63237fe640 4 set_mon_vals no callback set 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f6321ffb640 10 monclient: dump: 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:36:58.483 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 monclient: _renew_subs 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633415af70 con 0x7f63340c7c90 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 1 librados: init done 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:58.484 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:58.485 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.485 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.469+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6334152a30 con 0x7f63340c7c90 2026-02-22T21:36:58.582 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.580+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f63280413b0 con 0x7f63340c7c90 2026-02-22T21:36:58.583 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.581+0000 7f6321ffb640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 0x7f630407b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:36:58.583 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.581+0000 7f633967d640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 0x7f630407b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:36:58.583 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.581+0000 7f633967d640 10 monclient: get_auth_request con 0x7f6304078d70 auth_method 0 2026-02-22T21:36:58.583 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.581+0000 7f6321ffb640 10 client.4299.objecter ms_handle_connect 0x7f6304078d70 2026-02-22T21:36:58.583 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.581+0000 7f633967d640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 0x7f630407b230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6324002770 tx=0x7f6324007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f632802fb50 con 0x7f63340c7c90 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 10 client.4299.objecter ms_dispatch2 0x7f6334000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 3 client.4299.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 3 client.4299.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 20 client.4299.objecter dump_active .. 0 homeless 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6328086fe0 con 0x7f63340c7c90 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:36:58.591 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.589+0000 7f6321ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:36:58.679 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.677+0000 7f633b908640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:58.679 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.677+0000 7f633b908640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:58.679 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.677+0000 7f633b908640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:36:58.679 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.677+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63340c7740 con 0x7f63340c7c90 2026-02-22T21:36:58.689 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.687+0000 7f6321ffb640 1 -- 10.20.193.51:0/2129677627 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f632808b2e0 con 0x7f63340c7c90 2026-02-22T21:36:58.689 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.687+0000 7f6321ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:36:58.689 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.687+0000 7f6321ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:36:58.689 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:36:58.689 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 10 librados: watch_flush enter 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 10 librados: watch_flush exit 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 20 client.4299.objecter shutdown clearing up homeless session... 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 10 client.4299.objecter successfully canceled tick 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 msgr2=0x7f630407b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.692 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 0x7f630407b230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6324002770 tx=0x7f6324007ef0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 10 monclient: shutdown 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 msgr2=0x7f63341f63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f632802f430 tx=0x7f63280047c0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 shutdown_connections 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6304078d70 0x7f630407b230 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6334152650 0x7f63341f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f63340c7c90 0x7f63341f63d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 --2- 10.20.193.51:0/2129677627 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f633406fcc0 0x7f6334151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 >> 10.20.193.51:0/2129677627 conn(0x7f63340cd4f0 msgr2=0x7f633414cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 shutdown_connections 2026-02-22T21:36:58.693 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 -- 10.20.193.51:0/2129677627 wait complete. 2026-02-22T21:36:58.694 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:36:58.689+0000 7f633b908640 1 librados: shutdown 2026-02-22T21:36:58.698 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:37:00.849 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:32.003 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.852 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:32.006 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.870 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:32.024 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.883 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:32.035 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.884 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:32.000 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.885 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:32.036 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.886 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:32.002 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.902 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:32.054 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:00.905 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:32.021 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:06.310 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.366+0000 7f2248753640 10 monclient: get_monmap_and_config 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.366+0000 7f2248753640 10 monclient: build_initial_monmap 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.366+0000 7f2248753640 10 monclient: monmap: 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:06.368 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:37:06.367499+0000 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:37:06.367499+0000 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:06.369 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.371 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.367+0000 7f2248753640 1 Processor -- start 2026-02-22T21:37:06.371 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.367+0000 7f2248753640 1 -- start start 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.367+0000 7f2248753640 10 monclient: init 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.367+0000 7f2248753640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.367+0000 7f2248753640 10 monclient: _add_conns ranks=[2,3,4,1,0,5] 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f224015f590 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f224006e190 0x7f224015f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): picked mon.noname-a con 0x7f224006e190 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f224015feb0 0x7f224015d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): picked mon.noname-b con 0x7f224015feb0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2240072a00 con 0x7f224015f590 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f224006e800 con 0x7f224006e190 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f2240072ce0 con 0x7f224015feb0 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:06.372 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2248753640 10 monclient(hunting): authenticate will time out at 1101.948052s 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f224006e190 0x7f224015f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f224006e190 0x7f224015f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50686/0 (socket says 10.20.193.51:50686) 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 1 -- 10.20.193.51:0/2775891760 learned_addr learned my addr 10.20.193.51:0/2775891760 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): get_auth_request con 0x7f224006e190 auth_method 0 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2246cc9640 1 --2- 10.20.193.51:0/2775891760 >> v2:10.20.193.68:3300/0 conn(0x7f224015feb0 0x7f224015d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:06.373 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2246cc9640 10 monclient(hunting): get_auth_request con 0x7f224015feb0 auth_method 0 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2246cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2246cc9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.368+0000 7f2246cc9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.369+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.369+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.369+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 1 -- 10.20.193.51:0/2775891760 >> v2:10.20.193.68:3300/0 conn(0x7f224015feb0 msgr2=0x7f224015d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:06.374 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 1 --2- 10.20.193.51:0/2775891760 >> v2:10.20.193.68:3300/0 conn(0x7f224015feb0 0x7f224015d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 1 -- 10.20.193.51:0/2775891760 >> v1:10.20.193.125:6789/0 conn(0x7f224015f590 legacy=0x7f2240074180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 10 monclient: _finish_hunting 0 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 1 monclient: found mon.noname-a 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.370+0000 7f2245cc7640 1 -- 10.20.193.51:0/2775891760 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2240072fc0 con 0x7f224006e190 2026-02-22T21:37:06.384 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.371+0000 7f2245cc7640 1 --2- 10.20.193.51:0/2775891760 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224006e190 0x7f224015f970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22400c7d80 tx=0x7f223402eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62def00ca97b36a server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:06.384 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 1 -- 10.20.193.51:0/2775891760 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223403c070 con 0x7f224006e190 2026-02-22T21:37:06.384 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: dump: 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:06.385 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: _finish_auth 0 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: _check_auth_tickets 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f2248753640 5 monclient: authenticate success, global_id 4302 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f2248753640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:06.386 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 1 -- 10.20.193.51:0/2775891760 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223402fb40 con 0x7f224006e190 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 1 -- 10.20.193.51:0/2775891760 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234037440 con 0x7f224006e190 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f22454c6640 10 monclient: dump: 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:06.387 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f2248753640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.373+0000 7f2248753640 4 set_mon_vals no callback set 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 10 monclient: shutdown 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- 10.20.193.51:0/2775891760 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224006e190 msgr2=0x7f224015f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 --2- 10.20.193.51:0/2775891760 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224006e190 0x7f224015f970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22400c7d80 tx=0x7f223402eed0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- 10.20.193.51:0/2775891760 shutdown_connections 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 --2- 10.20.193.51:0/2775891760 >> v2:10.20.193.68:3300/0 conn(0x7f224015feb0 0x7f224015d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 --2- 10.20.193.51:0/2775891760 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224006e190 0x7f224015f970 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- 10.20.193.51:0/2775891760 >> 10.20.193.51:0/2775891760 conn(0x7f22400ca1e0 msgr2=0x7f22400ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:06.388 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- 10.20.193.51:0/2775891760 shutdown_connections 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- 10.20.193.51:0/2775891760 wait complete. 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 10 monclient: build_initial_monmap 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 10 monclient: monmap: 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:37:06.389 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 librados: starting msgr at 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 librados: starting objecter 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 Processor -- start 2026-02-22T21:37:06.390 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 -- start start 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 librados: setting wanted keys 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 1 librados: calling monclient init 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.374+0000 7f2248753640 10 monclient: init 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f224006e190 0x7f22401f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): picked mon.noname-c con 0x7f224006e190 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:06.391 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f224015f590 0x7f2240145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): picked mon.noname-b con 0x7f224015f590 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): picked mon.noname-a con 0x7f224015feb0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f224015d760 con 0x7f224015feb0 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2240073580 con 0x7f224015f590 2026-02-22T21:37:06.392 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f224015d8e0 con 0x7f224006e190 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2248753640 10 monclient(hunting): authenticate will time out at 1101.955263s 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50694/0 (socket says 10.20.193.51:50694) 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 1 -- 10.20.193.51:0/727772989 learned_addr learned my addr 10.20.193.51:0/727772989 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:06.393 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2245cc7640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f224015f590 0x7f2240145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): get_auth_request con 0x7f224015feb0 auth_method 0 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2245cc7640 10 monclient(hunting): get_auth_request con 0x7f224015f590 auth_method 0 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2245cc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2245cc7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:06.394 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2245cc7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f22464c8640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f224006e190 0x7f22401f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.375+0000 7f2246cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): get_auth_request con 0x7f224006e190 auth_method 0 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:06.395 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f2245cc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.376+0000 7f22464c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 -- 10.20.193.51:0/727772989 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f224006e190 msgr2=0x7f22401f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:06.396 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f224006e190 0x7f22401f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 -- 10.20.193.51:0/727772989 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f224015f590 msgr2=0x7f2240145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f224015f590 0x7f2240145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 10 monclient: _finish_hunting 0 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 monclient: found mon.noname-a 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.397 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.377+0000 7f2246cc9640 1 -- 10.20.193.51:0/727772989 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22400cc330 con 0x7f224015feb0 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.378+0000 7f2246cc9640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d30 tx=0x7f223c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.378+0000 7f2226ffd640 10 client.?.objecter ms_handle_connect 0x7f224015feb0 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.378+0000 7f2226ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c018020 con 0x7f224015feb0 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: dump: 2026-02-22T21:37:06.398 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:06.399 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 5 monclient: authenticate success, global_id 4305 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223c014480 con 0x7f224015feb0 2026-02-22T21:37:06.400 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c014870 con 0x7f224015feb0 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2244cc5640 4 set_mon_vals no callback set 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2226ffd640 10 monclient: dump: 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:06.401 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _renew_subs 2026-02-22T21:37:06.402 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22401f3ef0 con 0x7f224015feb0 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _renew_subs 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401f44b0 con 0x7f224015feb0 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 1 librados: init done 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:06.403 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:06.404 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.404 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.381+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22401554e0 con 0x7f224015feb0 2026-02-22T21:37:06.405 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.394+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f223c014a10 con 0x7f224015feb0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.394+0000 7f2226ffd640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 0x7f221007b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.394+0000 7f22464c8640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 0x7f221007b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.394+0000 7f22464c8640 10 monclient: get_auth_request con 0x7f2210078dc0 auth_method 0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.395+0000 7f2226ffd640 10 client.4305.objecter ms_handle_connect 0x7f2210078dc0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.395+0000 7f22464c8640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 0x7f221007b280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22300046e0 tx=0x7f22300091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.396+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7f223c05ed50 con 0x7f224015feb0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.396+0000 7f2226ffd640 10 client.4305.objecter ms_dispatch2 0x7f2240000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.396+0000 7f2226ffd640 3 client.4305.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.396+0000 7f2226ffd640 3 client.4305.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:37:06.406 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.396+0000 7f2226ffd640 20 client.4305.objecter dump_active .. 0 homeless 2026-02-22T21:37:06.507 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.505+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f22401554e0 con 0x7f224015feb0 2026-02-22T21:37:06.507 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.505+0000 7f2226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:06.507 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.505+0000 7f2226ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:06.596 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.595+0000 7f2248753640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:06.596 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.595+0000 7f2248753640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:06.596 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.595+0000 7f2248753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:06.597 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.595+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f224015f970 con 0x7f224015feb0 2026-02-22T21:37:06.607 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.606+0000 7f2226ffd640 1 -- 10.20.193.51:0/727772989 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7f223c022070 con 0x7f224015feb0 2026-02-22T21:37:06.607 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.606+0000 7f2226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:06.607 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.606+0000 7f2226ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:06.608 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:37:06.608 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 10 librados: watch_flush enter 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 10 librados: watch_flush exit 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 20 client.4305.objecter shutdown clearing up homeless session... 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 10 client.4305.objecter successfully canceled tick 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 msgr2=0x7f221007b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 0x7f221007b280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22300046e0 tx=0x7f22300091c0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.610 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 10 monclient: shutdown 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 msgr2=0x7f224015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d30 tx=0x7f223c00c490 comp rx=0 tx=0).stop 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 shutdown_connections 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2210078dc0 0x7f221007b280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f224015feb0 0x7f224015e5c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f224015f590 0x7f2240145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 --2- 10.20.193.51:0/727772989 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f224006e190 0x7f22401f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 >> 10.20.193.51:0/727772989 conn(0x7f22400ca1e0 msgr2=0x7f22401e97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 shutdown_connections 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 -- 10.20.193.51:0/727772989 wait complete. 2026-02-22T21:37:06.611 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:06.608+0000 7f2248753640 1 librados: shutdown 2026-02-22T21:37:06.617 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:37:14.231 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:14.293 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.291+0000 7fbf1de7c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:14.293 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.291+0000 7fbf1de7c640 10 monclient: build_initial_monmap 2026-02-22T21:37:14.293 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.291+0000 7fbf1de7c640 10 monclient: monmap: 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:37:14.292247+0000 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:37:14.292247+0000 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:14.294 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:14.295 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:14.295 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.295 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.291+0000 7fbf1de7c640 1 Processor -- start 2026-02-22T21:37:14.295 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 1 -- start start 2026-02-22T21:37:14.295 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient: init 2026-02-22T21:37:14.297 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:14.297 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient: _add_conns ranks=[2,5,0,3,4,1] 2026-02-22T21:37:14.297 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf1806ff30 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fbf18070310 0x7fbf18159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-c con 0x7fbf18070310 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf1815a000 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf180c5570 con 0x7fbf1815a000 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf180c5290 con 0x7fbf1806ff30 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fbf1815db10 con 0x7fbf18070310 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.292+0000 7fbf1de7c640 10 monclient(hunting): authenticate will time out at 1109.872553s 2026-02-22T21:37:14.298 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf17fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fbf1815a000 0x7fbf180c4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:33552/0 (socket says 10.20.193.51:33552) 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf17fff640 1 -- 10.20.193.51:0/289951395 learned_addr learned my addr 10.20.193.51:0/289951395 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf16ffd640 1 --2- 10.20.193.51:0/289951395 >> v2:10.20.193.125:3300/0 conn(0x7fbf18070310 0x7fbf18159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf16ffd640 10 monclient(hunting): get_auth_request con 0x7fbf18070310 auth_method 0 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf16ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.293+0000 7fbf16ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.294+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.294+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.294+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 1 -- 10.20.193.51:0/289951395 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192040220 0 0) 0x7fbf0c002ca0 con 0x7fbf1815a000 2026-02-22T21:37:14.299 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 10 monclient(hunting): my global_id is 4308 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf167fc640 1 -- 10.20.193.51:0/289951395 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbefc003620 con 0x7fbf1815a000 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 1 -- 10.20.193.51:0/289951395 >> v1:10.20.193.125:6789/0 conn(0x7fbf1806ff30 legacy=0x7fbf180c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 1 -- 10.20.193.51:0/289951395 >> v1:10.20.193.51:6789/0 conn(0x7fbf1815a000 legacy=0x7fbf180c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 1 monclient: found mon.noname-c 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:14.300 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.301 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.295+0000 7fbf16ffd640 1 -- 10.20.193.51:0/289951395 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1815d1e0 con 0x7fbf18070310 2026-02-22T21:37:14.305 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.298+0000 7fbf16ffd640 1 --2- 10.20.193.51:0/289951395 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf18070310 0x7fbf18159ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf1815dc90 tx=0x7fbf0802eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3e7a952ec9fe541 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 1 -- 10.20.193.51:0/289951395 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0803c070 con 0x7fbf18070310 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: dump: 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:14.306 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 1 -- 10.20.193.51:0/289951395 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf0802fbf0 con 0x7fbf18070310 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 5 monclient: authenticate success, global_id 4307 2026-02-22T21:37:14.307 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 1 -- 10.20.193.51:0/289951395 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08037440 con 0x7fbf18070310 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 4 set_mon_vals no callback set 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf167fc640 10 monclient: dump: 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.308 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 10 monclient: shutdown 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 -- 10.20.193.51:0/289951395 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf18070310 msgr2=0x7fbf18159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/289951395 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf18070310 0x7fbf18159ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf1815dc90 tx=0x7fbf0802eee0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 -- 10.20.193.51:0/289951395 shutdown_connections 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/289951395 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf18070310 0x7fbf18159ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 -- 10.20.193.51:0/289951395 >> 10.20.193.51:0/289951395 conn(0x7fbf180cd4f0 msgr2=0x7fbf180cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 -- 10.20.193.51:0/289951395 shutdown_connections 2026-02-22T21:37:14.309 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.300+0000 7fbf1de7c640 1 -- 10.20.193.51:0/289951395 wait complete. 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient: build_initial_monmap 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient: monmap: 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:14.310 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 librados: starting msgr at 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 librados: starting objecter 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 Processor -- start 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 -- start start 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 librados: setting wanted keys 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 librados: calling monclient init 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient: init 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:14.311 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-c con 0x7fbf1806ff30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbf18070310 0x7fbf1813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-a con 0x7fbf18070310 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbf1815a000 0x7fbf18148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): picked mon.noname-b con 0x7fbf1815a000 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbf180c99e0 con 0x7fbf18070310 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1815d7a0 con 0x7fbf1815a000 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbf180c4e20 con 0x7fbf1806ff30 2026-02-22T21:37:14.312 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.301+0000 7fbf1de7c640 10 monclient(hunting): authenticate will time out at 1109.881705s 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbf18070310 0x7fbf1813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:54798/0 (socket says 10.20.193.51:54798) 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 1 -- 10.20.193.51:0/991810464 learned_addr learned my addr 10.20.193.51:0/991810464 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): get_auth_request con 0x7fbf1806ff30 auth_method 0 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): get_auth_request con 0x7fbf18070310 auth_method 0 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:14.313 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf17fff640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbf1815a000 0x7fbf18148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf17fff640 10 monclient(hunting): get_auth_request con 0x7fbf1815a000 auth_method 0 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf17fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf17fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:14.314 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.302+0000 7fbf16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.303+0000 7fbf17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.303+0000 7fbf17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.303+0000 7fbf17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 -- 10.20.193.51:0/991810464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbf1815a000 msgr2=0x7fbf18148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbf1815a000 0x7fbf18148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 -- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbf18070310 msgr2=0x7fbf1813f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbf18070310 0x7fbf1813f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:14.315 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.316 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.304+0000 7fbf177fe640 1 -- 10.20.193.51:0/991810464 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf18149350 con 0x7fbf1806ff30 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.305+0000 7fbf177fe640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c5b0 tx=0x7fbf0400ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.305+0000 7fbef3fff640 10 client.?.objecter ms_handle_connect 0x7fbf1806ff30 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.305+0000 7fbef3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040052d0 con 0x7fbf1806ff30 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: dump: 2026-02-22T21:37:14.318 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:14.319 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf04005470 con 0x7fbf1806ff30 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040038f0 con 0x7fbf1806ff30 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbef3fff640 10 monclient: dump: 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:14.320 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf15ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 5 monclient: authenticate success, global_id 4313 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 10 monclient: _renew_subs 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf180be930 con 0x7fbf1806ff30 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:14.321 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 10 monclient: _renew_subs 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf18149070 con 0x7fbf1806ff30 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.307+0000 7fbf1de7c640 1 librados: init done 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.308+0000 7fbf1de7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.308+0000 7fbf1de7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.308+0000 7fbf1de7c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.322 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.308+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf18159ac0 con 0x7fbf1806ff30 2026-02-22T21:37:14.323 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.319+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7fbf04003a90 con 0x7fbf1806ff30 2026-02-22T21:37:14.323 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.320+0000 7fbef3fff640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 0x7fbee807b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:14.323 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.320+0000 7fbf16ffd640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 0x7fbee807b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:14.323 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.320+0000 7fbf16ffd640 10 monclient: get_auth_request con 0x7fbee8078df0 auth_method 0 2026-02-22T21:37:14.323 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.320+0000 7fbef3fff640 10 client.4313.objecter ms_handle_connect 0x7fbee8078df0 2026-02-22T21:37:14.324 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.320+0000 7fbf16ffd640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 0x7fbee807b2b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf08039000 tx=0x7fbf08033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:14.324 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.322+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7fbf0405dfd0 con 0x7fbf1806ff30 2026-02-22T21:37:14.324 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.322+0000 7fbef3fff640 10 client.4313.objecter ms_dispatch2 0x7fbf18000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:37:14.324 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.322+0000 7fbef3fff640 3 client.4313.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:37:14.325 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.322+0000 7fbef3fff640 3 client.4313.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:37:14.325 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.322+0000 7fbef3fff640 20 client.4313.objecter dump_active .. 0 homeless 2026-02-22T21:37:14.429 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.428+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fbf04061f90 con 0x7fbf1806ff30 2026-02-22T21:37:14.429 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.428+0000 7fbef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:14.429 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.428+0000 7fbef3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:14.519 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.516+0000 7fbf1de7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:14.519 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.516+0000 7fbf1de7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:14.519 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.516+0000 7fbf1de7c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:14.519 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.516+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf1815a790 con 0x7fbf1806ff30 2026-02-22T21:37:14.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.552+0000 7fbef3fff640 1 -- 10.20.193.51:0/991810464 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7fbf0407b260 con 0x7fbf1806ff30 2026-02-22T21:37:14.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.552+0000 7fbef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:14.554 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.552+0000 7fbef3fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:14.554 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:37:14.555 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 10 librados: watch_flush enter 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 10 librados: watch_flush exit 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 20 client.4313.objecter shutdown clearing up homeless session... 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 10 client.4313.objecter successfully canceled tick 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 msgr2=0x7fbee807b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 0x7fbee807b2b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf08039000 tx=0x7fbf08033000 comp rx=0 tx=0).stop 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 10 monclient: shutdown 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:14.557 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 msgr2=0x7fbf180cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c5b0 tx=0x7fbf0400ca80 comp rx=0 tx=0).stop 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 shutdown_connections 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbee8078df0 0x7fbee807b2b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbf1815a000 0x7fbf18148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbf18070310 0x7fbf1813f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 --2- 10.20.193.51:0/991810464 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbf1806ff30 0x7fbf180cb0f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 >> 10.20.193.51:0/991810464 conn(0x7fbf180cd4f0 msgr2=0x7fbf181591f0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 shutdown_connections 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 -- 10.20.193.51:0/991810464 wait complete. 2026-02-22T21:37:14.558 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:14.554+0000 7fbf1de7c640 1 librados: shutdown 2026-02-22T21:37:14.564 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:37:20.882 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:52.035 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.884 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:52.038 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.891 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:01:52.045 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.914 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:52.066 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.915 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:52.031 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.916 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:52.067 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.916 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:52.032 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.925 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:01:52.076 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:20.925 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:01:52.041 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:22.177 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:22.239 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.236+0000 7ffac5a16640 10 monclient: get_monmap_and_config 2026-02-22T21:37:22.239 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.236+0000 7ffac5a16640 10 monclient: build_initial_monmap 2026-02-22T21:37:22.239 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.236+0000 7ffac5a16640 10 monclient: monmap: 2026-02-22T21:37:22.239 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:37:22.238262+0000 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:37:22.238262+0000 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:22.240 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:22.241 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.241 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 Processor -- start 2026-02-22T21:37:22.241 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 -- start start 2026-02-22T21:37:22.241 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient: init 2026-02-22T21:37:22.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:22.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient: _add_conns ranks=[0,3,4,1,5,2] 2026-02-22T21:37:22.243 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffac006ff30 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ffac0070310 0x7ffac0159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-a con 0x7ffac0070310 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ffac015a000 0x7ffac00c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-b con 0x7ffac015a000 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffac00c5570 con 0x7ffac006ff30 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ffac015db10 con 0x7ffac0070310 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7ffac00c5290 con 0x7ffac015a000 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:22.244 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.237+0000 7ffac5a16640 10 monclient(hunting): authenticate will time out at 1117.818581s 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabe7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ffac0070310 0x7ffac0159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabf7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ffac015a000 0x7ffac00c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabeffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7ffac006ff30 0x7ffac00c92d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:36184/0 (socket says 10.20.193.51:36184) 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabeffd640 1 -- 10.20.193.51:0/1472549215 learned_addr learned my addr 10.20.193.51:0/1472549215 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabe7fc640 10 monclient(hunting): get_auth_request con 0x7ffac0070310 auth_method 0 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabe7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.238+0000 7ffabe7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): get_auth_request con 0x7ffac015a000 auth_method 0 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:22.245 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.239+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2026-02-22T21:37:22.246 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 1 -- 10.20.193.51:0/1472549215 >> v2:10.20.193.68:3300/0 conn(0x7ffac015a000 msgr2=0x7ffac00c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 1 --2- 10.20.193.51:0/1472549215 >> v2:10.20.193.68:3300/0 conn(0x7ffac015a000 0x7ffac00c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 1 -- 10.20.193.51:0/1472549215 >> v1:10.20.193.51:6789/0 conn(0x7ffac006ff30 legacy=0x7ffac00c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 1 monclient: found mon.noname-a 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.247 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.240+0000 7ffabe7fc640 1 -- 10.20.193.51:0/1472549215 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac015d1e0 con 0x7ffac0070310 2026-02-22T21:37:22.252 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.243+0000 7ffabe7fc640 1 --2- 10.20.193.51:0/1472549215 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac0070310 0x7ffac0159ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffac006f3f0 tx=0x7ffaa802eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8f2e0305ef34d0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:22.252 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 1 -- 10.20.193.51:0/1472549215 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa803c070 con 0x7ffac0070310 2026-02-22T21:37:22.252 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:22.252 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:22.252 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: dump: 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.253 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffac5a16640 5 monclient: authenticate success, global_id 4314 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 1 -- 10.20.193.51:0/1472549215 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa802fb80 con 0x7ffac0070310 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffac5a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffabdffb640 1 -- 10.20.193.51:0/1472549215 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8037440 con 0x7ffac0070310 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.244+0000 7ffac5a16640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 4 set_mon_vals no callback set 2026-02-22T21:37:22.254 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffabdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffabdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffabdffb640 10 monclient: dump: 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:22.255 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 10 monclient: shutdown 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 -- 10.20.193.51:0/1472549215 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac0070310 msgr2=0x7ffac0159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1472549215 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac0070310 0x7ffac0159ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffac006f3f0 tx=0x7ffaa802eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 -- 10.20.193.51:0/1472549215 shutdown_connections 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1472549215 >> v2:10.20.193.68:3300/0 conn(0x7ffac015a000 0x7ffac00c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.256 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1472549215 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac0070310 0x7ffac0159ac0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 -- 10.20.193.51:0/1472549215 >> 10.20.193.51:0/1472549215 conn(0x7ffac00cd4f0 msgr2=0x7ffac00cd900 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 -- 10.20.193.51:0/1472549215 shutdown_connections 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.245+0000 7ffac5a16640 1 -- 10.20.193.51:0/1472549215 wait complete. 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient: build_initial_monmap 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient: monmap: 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:22.257 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 librados: starting msgr at 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 librados: starting objecter 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 Processor -- start 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 -- start start 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 librados: setting wanted keys 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 librados: calling monclient init 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient: init 2026-02-22T21:37:22.258 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffac006ff30 0x7ffac00c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-c con 0x7ffac006ff30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffac0070310 0x7ffac01f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-b con 0x7ffac0070310 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): picked mon.noname-a con 0x7ffac015a000 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00c99e0 con 0x7ffac015a000 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:22.259 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ffac015d7a0 con 0x7ffac0070310 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00c4c50 con 0x7ffac006ff30 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.246+0000 7ffac5a16640 10 monclient(hunting): authenticate will time out at 1117.827907s 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48518/0 (socket says 10.20.193.51:48518) 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 1 -- 10.20.193.51:0/1056701615 learned_addr learned my addr 10.20.193.51:0/1056701615 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): get_auth_request con 0x7ffac015a000 auth_method 0 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:22.260 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabe7fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffac0070310 0x7ffac01f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabeffd640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffac006ff30 0x7ffac00c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabe7fc640 10 monclient(hunting): get_auth_request con 0x7ffac0070310 auth_method 0 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabe7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabe7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:22.261 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabeffd640 10 monclient(hunting): get_auth_request con 0x7ffac006ff30 auth_method 0 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.247+0000 7ffabeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.248+0000 7ffabeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 -- 10.20.193.51:0/1056701615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffac006ff30 msgr2=0x7ffac00c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.262 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffac006ff30 0x7ffac00c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 -- 10.20.193.51:0/1056701615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffac0070310 msgr2=0x7ffac01f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffac0070310 0x7ffac01f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.263 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.249+0000 7ffabf7fe640 1 -- 10.20.193.51:0/1056701615 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac01f3ac0 con 0x7ffac015a000 2026-02-22T21:37:22.265 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.250+0000 7ffabf7fe640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009d30 tx=0x7ffab000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:22.265 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.250+0000 7ffa97fff640 10 client.?.objecter ms_handle_connect 0x7ffac015a000 2026-02-22T21:37:22.265 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.250+0000 7ffa97fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:22.265 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018020 con 0x7ffac015a000 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 10 monclient: dump: 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:22.266 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffa97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 5 monclient: authenticate success, global_id 4320 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 10 monclient: _renew_subs 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.267 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac01f3da0 con 0x7ffac015a000 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 10 monclient: _renew_subs 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac01f4360 con 0x7ffac015a000 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.251+0000 7ffac5a16640 1 librados: init done 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffac5a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffac5a16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffac5a16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac0159ac0 con 0x7ffac015a000 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab0014480 con 0x7ffac015a000 2026-02-22T21:37:22.268 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0014830 con 0x7ffac015a000 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffabd7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.252+0000 7ffa97fff640 10 monclient: dump: 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:22.269 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:22.270 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:22.270 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:22.365 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.363+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7ffab00149d0 con 0x7ffac015a000 2026-02-22T21:37:22.366 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.363+0000 7ffa97fff640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 0x7ffa9007b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:22.366 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.363+0000 7ffabeffd640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 0x7ffa9007b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:22.366 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.363+0000 7ffabeffd640 10 monclient: get_auth_request con 0x7ffa90078d70 auth_method 0 2026-02-22T21:37:22.367 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.364+0000 7ffabeffd640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 0x7ffa9007b230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffab4002770 tx=0x7ffab4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:22.367 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.364+0000 7ffa97fff640 10 client.4320.objecter ms_handle_connect 0x7ffa90078d70 2026-02-22T21:37:22.378 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2565+0+0 (secure 0 0 0) 0x7ffab005f0d0 con 0x7ffac015a000 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 10 client.4320.objecter ms_dispatch2 0x7ffac0000b90 osd_map(18..18 src has 1..18) 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 3 client.4320.objecter handle_osd_map got epochs [18,18] > 0 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 3 client.4320.objecter handle_osd_map decoding full epoch 18 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 20 client.4320.objecter dump_active .. 0 homeless 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ffab0022070 con 0x7ffac015a000 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:22.379 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.376+0000 7ffa97fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:22.466 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.463+0000 7ffac5a16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:22.466 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.463+0000 7ffac5a16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:22.466 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.463+0000 7ffac5a16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:22.466 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.463+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffac00c8d20 con 0x7ffac015a000 2026-02-22T21:37:22.476 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.474+0000 7ffa97fff640 1 -- 10.20.193.51:0/1056701615 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12879 (secure 0 0 0) 0x7ffab0010040 con 0x7ffac015a000 2026-02-22T21:37:22.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.474+0000 7ffa97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:22.477 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.474+0000 7ffa97fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:22.477 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:37:22.477 INFO:teuthology.misc.health.trial051.stdout:{"epoch":18,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:35:52.985822+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 10 librados: watch_flush enter 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 10 librados: watch_flush exit 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 20 client.4320.objecter shutdown clearing up homeless session... 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 10 client.4320.objecter successfully canceled tick 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 msgr2=0x7ffa9007b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.480 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 0x7ffa9007b230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffab4002770 tx=0x7ffab4009040 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 10 monclient: shutdown 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 msgr2=0x7ffac01f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.476+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009d30 tx=0x7ffab000c490 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 shutdown_connections 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffa90078d70 0x7ffa9007b230 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffac015a000 0x7ffac01f56f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffac0070310 0x7ffac01f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 --2- 10.20.193.51:0/1056701615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffac006ff30 0x7ffac00c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 >> 10.20.193.51:0/1056701615 conn(0x7ffac00cd4f0 msgr2=0x7ffac00be800 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 shutdown_connections 2026-02-22T21:37:22.481 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 -- 10.20.193.51:0/1056701615 wait complete. 2026-02-22T21:37:22.482 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:22.477+0000 7ffac5a16640 1 librados: shutdown 2026-02-22T21:37:22.487 DEBUG:teuthology.misc:0 of 16 OSDs are up 2026-02-22T21:37:23.192 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:37:23.186+0000 77b8940 -1 osd.15 0 log_to_monitors true 2026-02-22T21:37:23.754 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:37:23.748+0000 77b8940 -1 osd.7 0 log_to_monitors true 2026-02-22T21:37:24.433 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:37:24.427+0000 77b8940 -1 osd.11 0 log_to_monitors true 2026-02-22T21:37:24.557 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:37:24.551+0000 77b8940 -1 osd.3 0 log_to_monitors true 2026-02-22T21:37:24.822 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:37:24.818+0000 77b8940 -1 osd.14 0 log_to_monitors true 2026-02-22T21:37:24.955 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:37:24.948+0000 77b8940 -1 osd.10 0 log_to_monitors true 2026-02-22T21:37:25.125 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:37:25.118+0000 77b8940 -1 osd.0 0 log_to_monitors true 2026-02-22T21:37:25.209 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:37:25.204+0000 d2b0640 -1 osd.15 0 waiting for initial osdmap 2026-02-22T21:37:25.240 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:37:25.233+0000 77b8940 -1 osd.5 0 log_to_monitors true 2026-02-22T21:37:25.243 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:37:25.239+0000 77b8940 -1 osd.2 0 log_to_monitors true 2026-02-22T21:37:25.243 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:37:25.239+0000 77b8940 -1 osd.4 0 log_to_monitors true 2026-02-22T21:37:25.303 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:37:25.296+0000 77b8940 -1 osd.13 0 log_to_monitors true 2026-02-22T21:37:25.429 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:37:25.419+0000 11c76640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:25.451 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:37:25.445+0000 77b8940 -1 osd.6 0 log_to_monitors true 2026-02-22T21:37:25.536 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:37:25.529+0000 77b8940 -1 osd.9 0 log_to_monitors true 2026-02-22T21:37:25.687 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:37:25.680+0000 77b8940 -1 osd.12 0 log_to_monitors true 2026-02-22T21:37:25.876 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:37:25.872+0000 77b8940 -1 osd.8 0 log_to_monitors true 2026-02-22T21:37:26.271 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:37:26.266+0000 d2b0640 -1 osd.7 0 waiting for initial osdmap 2026-02-22T21:37:26.304 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:37:26.298+0000 dab1640 -1 osd.11 0 waiting for initial osdmap 2026-02-22T21:37:26.403 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:37:26.396+0000 77b8940 -1 osd.1 0 log_to_monitors true 2026-02-22T21:37:26.492 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:37:26.481+0000 11c76640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:26.527 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:37:26.517+0000 15c7e640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.273 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:37:27.266+0000 d2b0640 -1 osd.13 0 waiting for initial osdmap 2026-02-22T21:37:27.280 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:37:27.272+0000 dab1640 -1 osd.5 0 waiting for initial osdmap 2026-02-22T21:37:27.301 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:37:27.298+0000 dab1640 -1 osd.6 0 waiting for initial osdmap 2026-02-22T21:37:27.310 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:37:27.305+0000 dab1640 -1 osd.4 0 waiting for initial osdmap 2026-02-22T21:37:27.334 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:37:27.326+0000 dab1640 -1 osd.9 0 waiting for initial osdmap 2026-02-22T21:37:27.373 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:37:27.367+0000 dab1640 -1 osd.3 0 waiting for initial osdmap 2026-02-22T21:37:27.373 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:37:27.370+0000 dab1640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T21:37:27.374 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:37:27.369+0000 dab1640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T21:37:27.374 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:37:27.370+0000 dab1640 -1 osd.10 0 waiting for initial osdmap 2026-02-22T21:37:27.374 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:37:27.370+0000 d2b0640 -1 osd.14 0 waiting for initial osdmap 2026-02-22T21:37:27.537 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:37:27.529+0000 15c7e640 -1 osd.6 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.567 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:37:27.555+0000 15c7e640 -1 osd.5 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.569 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:37:27.558+0000 15c7e640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.578 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:37:27.569+0000 15c7e640 -1 osd.10 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.578 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:37:27.570+0000 11c76640 -1 osd.2 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.579 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:37:27.567+0000 15c7e640 -1 osd.9 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.581 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:37:27.573+0000 15c7e640 -1 osd.4 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.584 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:37:27.576+0000 15c7e640 -1 osd.14 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.590 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:37:27.580+0000 15c7e640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:27.617 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:37:27.608+0000 15c7e640 -1 osd.0 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:28.347 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:37:28.344+0000 d2b0640 -1 osd.12 0 waiting for initial osdmap 2026-02-22T21:37:28.369 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:37:28.365+0000 dab1640 -1 osd.8 0 waiting for initial osdmap 2026-02-22T21:37:28.461 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:37:28.454+0000 dab1640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T21:37:28.644 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:37:28.635+0000 11c76640 -1 osd.8 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:28.651 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:37:28.642+0000 15c7e640 -1 osd.12 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:28.695 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:37:28.683+0000 15c7e640 -1 osd.1 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T21:37:30.098 DEBUG:teuthology.orchestra.run.trial051:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:30.159 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: get_monmap_and_config 2026-02-22T21:37:30.159 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: build_initial_monmap 2026-02-22T21:37:30.159 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: monmap: 2026-02-22T21:37:30.159 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:30.159 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:37:30.156711+0000 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:37:30.156711+0000 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:30.160 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 Processor -- start 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 -- start start 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: init 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient: _add_conns ranks=[0,4,1,3,5,2] 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c7415f480 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f0c7415f860 0x7f0c740c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-b con 0x7f0c7415f860 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f0c74151ff0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c740c9f50 con 0x7f0c7415f480 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c740c8bd0 con 0x7f0c74151ff0 2026-02-22T21:37:30.161 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7415fcb0 con 0x7f0c7415f860 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.155+0000 7f0c7c888640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.156+0000 7f0c7c888640 10 monclient(hunting): authenticate will time out at 1125.737004s 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.156+0000 7f0c7a5fd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f0c7415f480 0x7f0c7415d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:50098/0 (socket says 10.20.193.51:50098) 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.156+0000 7f0c7a5fd640 1 -- 10.20.193.51:0/2953731379 learned_addr learned my addr 10.20.193.51:0/2953731379 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.156+0000 7f0c79dfc640 1 --2- 10.20.193.51:0/2953731379 >> v2:10.20.193.68:3300/0 conn(0x7f0c7415f860 0x7f0c740c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): get_auth_request con 0x7f0c7415f860 auth_method 0 2026-02-22T21:37:30.162 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.169 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 1 -- 10.20.193.51:0/2953731379 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219218651 0 0) 0x7f0c740c9f50 con 0x7f0c7415f480 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.170 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 10 monclient(hunting): my global_id is 4365 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.157+0000 7f0c795fb640 1 -- 10.20.193.51:0/2953731379 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c64003620 con 0x7f0c7415f480 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 1 -- 10.20.193.51:0/2953731379 >> v1:10.20.193.68:6789/0 conn(0x7f0c74151ff0 legacy=0x7f0c74152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 1 -- 10.20.193.51:0/2953731379 >> v1:10.20.193.51:6789/0 conn(0x7f0c7415f480 legacy=0x7f0c7415d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 1 monclient: found mon.noname-b 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.158+0000 7f0c79dfc640 1 -- 10.20.193.51:0/2953731379 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c740c8eb0 con 0x7f0c7415f860 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.160+0000 7f0c79dfc640 1 --2- 10.20.193.51:0/2953731379 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c7415f860 0x7f0c740c84e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c600098d0 tx=0x7f0c6002ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9573e5b3820da53b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.171 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 1 -- 10.20.193.51:0/2953731379 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6003c070 con 0x7f0c7415f860 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: dump: 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.172 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: _finish_auth 0 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 1 -- 10.20.193.51:0/2953731379 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6002fbf0 con 0x7f0c7415f860 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 1 -- 10.20.193.51:0/2953731379 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c60037440 con 0x7f0c7415f860 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 5 monclient: authenticate success, global_id 4363 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 4 set_mon_vals no callback set 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.173 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c795fb640 10 monclient: dump: 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.174 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 10 monclient: shutdown 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 -- 10.20.193.51:0/2953731379 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c7415f860 msgr2=0x7f0c740c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 --2- 10.20.193.51:0/2953731379 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c7415f860 0x7f0c740c84e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c600098d0 tx=0x7f0c6002ef20 comp rx=0 tx=0).stop 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 -- 10.20.193.51:0/2953731379 shutdown_connections 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 --2- 10.20.193.51:0/2953731379 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c7415f860 0x7f0c740c84e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 -- 10.20.193.51:0/2953731379 >> 10.20.193.51:0/2953731379 conn(0x7f0c7406c890 msgr2=0x7f0c7406cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 -- 10.20.193.51:0/2953731379 shutdown_connections 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.162+0000 7f0c7c888640 1 -- 10.20.193.51:0/2953731379 wait complete. 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient: build_initial_monmap 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient: monmap: 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:epoch 0 2026-02-22T21:37:30.175 INFO:teuthology.misc.health.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:created 0.000000 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 librados: starting msgr at 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 librados: starting objecter 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 Processor -- start 2026-02-22T21:37:30.176 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 -- start start 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 librados: setting wanted keys 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 librados: calling monclient init 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient: init 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 0x7f0c741e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-b con 0x7f0c74151ff0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 0x7f0c741538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-a con 0x7f0c7415f480 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c7415f860 0x7f0c74153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): picked mon.noname-c con 0x7f0c7415f860 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:30.177 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0c740c9750 con 0x7f0c7415f480 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f0c740c9470 con 0x7f0c74151ff0 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0c740c9a30 con 0x7f0c7415f860 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.163+0000 7f0c7c888640 10 monclient(hunting): authenticate will time out at 1125.744905s 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 0x7f0c741538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 0x7f0c741e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 0x7f0c741538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:55242/0 (socket says 10.20.193.51:55242) 2026-02-22T21:37:30.178 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 1 -- 10.20.193.51:0/4236544937 learned_addr learned my addr 10.20.193.51:0/4236544937 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): get_auth_request con 0x7f0c7415f480 auth_method 0 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): get_auth_request con 0x7f0c74151ff0 auth_method 0 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7adfe640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c7415f860 0x7f0c74153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.179 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7a5fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c79dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7adfe640 10 monclient(hunting): get_auth_request con 0x7f0c7415f860 auth_method 0 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7adfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7adfe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.164+0000 7f0c7adfe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.165+0000 7f0c7adfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.165+0000 7f0c7adfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.165+0000 7f0c7adfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2026-02-22T21:37:30.180 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 -- 10.20.193.51:0/4236544937 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c7415f860 msgr2=0x7f0c74153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c7415f860 0x7f0c74153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 -- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 msgr2=0x7f0c741538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 0x7f0c741538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 monclient: found mon.noname-b 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.166+0000 7f0c7a5fd640 1 -- 10.20.193.51:0/4236544937 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7406eab0 con 0x7f0c74151ff0 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.167+0000 7f0c7a5fd640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 0x7f0c741e98a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c4d0 tx=0x7f0c6c00c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.167+0000 7f0c5effd640 10 client.?.objecter ms_handle_connect 0x7f0c74151ff0 2026-02-22T21:37:30.181 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.167+0000 7f0c5effd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:30.183 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c017070 con 0x7f0c74151ff0 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 10 monclient: dump: 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.184 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 10 monclient: _finish_auth 0 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c7c888640 5 monclient: authenticate success, global_id 4369 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c7c888640 10 monclient: _renew_subs 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6c005de0 con 0x7f0c74151ff0 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c7c888640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.168+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7406ed70 con 0x7f0c74151ff0 2026-02-22T21:37:30.185 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c5effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c007290 con 0x7f0c74151ff0 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c5effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c5effd640 10 monclient: dump: 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:epoch 1 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.186 INFO:teuthology.misc.health.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr: 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c78dfa640 4 set_mon_vals no callback set 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 monclient: _renew_subs 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7406f330 con 0x7f0c74151ff0 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 1 librados: init done 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.187 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.169+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c740c84e0 con 0x7f0c74151ff0 2026-02-22T21:37:30.277 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.274+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f0c6c007430 con 0x7f0c74151ff0 2026-02-22T21:37:30.278 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.275+0000 7f0c5effd640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 0x7f0c4407b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.278 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.275+0000 7f0c79dfc640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 0x7f0c4407b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.278 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.275+0000 7f0c79dfc640 10 monclient: get_auth_request con 0x7f0c44078d70 auth_method 0 2026-02-22T21:37:30.278 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.275+0000 7f0c5effd640 10 client.4369.objecter ms_handle_connect 0x7f0c44078d70 2026-02-22T21:37:30.278 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.275+0000 7f0c79dfc640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 0x7f0c4407b230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c6002f450 tx=0x7f0c60033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.279 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.276+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 8563+0+0 (secure 0 0 0) 0x7f0c6c05d3f0 con 0x7f0c74151ff0 2026-02-22T21:37:30.279 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.276+0000 7f0c5effd640 10 client.4369.objecter ms_dispatch2 0x7f0c74000b90 osd_map(24..24 src has 1..24) 2026-02-22T21:37:30.279 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.276+0000 7f0c5effd640 3 client.4369.objecter handle_osd_map got epochs [24,24] > 0 2026-02-22T21:37:30.279 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.276+0000 7f0c5effd640 3 client.4369.objecter handle_osd_map decoding full epoch 24 2026-02-22T21:37:30.279 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.276+0000 7f0c5effd640 20 client.4369.objecter dump_active .. 0 homeless 2026-02-22T21:37:30.285 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.283+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0c6c021020 con 0x7f0c74151ff0 2026-02-22T21:37:30.285 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.283+0000 7f0c5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.285 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.283+0000 7f0c5effd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:30.375 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.373+0000 7f0c7c888640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:30.376 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.373+0000 7f0c7c888640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:30.376 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.373+0000 7f0c7c888640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.376 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.373+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c740c8910 con 0x7f0c74151ff0 2026-02-22T21:37:30.450 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.447+0000 7f0c5effd640 1 -- 10.20.193.51:0/4236544937 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+22722 (secure 0 0 0) 0x7f0c6c0613b0 con 0x7f0c74151ff0 2026-02-22T21:37:30.450 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.447+0000 7f0c5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:30.450 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.447+0000 7f0c5effd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:30.450 INFO:teuthology.misc.health.trial051.stdout: 2026-02-22T21:37:30.450 INFO:teuthology.misc.health.trial051.stdout:{"epoch":24,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:37:29.474866+0000","last_up_change":"2026-02-22T21:37:29.474866+0000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T21:37:27.901086+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6802","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6803","nonce":1198482343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6804","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6805","nonce":1198482343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6808","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6809","nonce":1198482343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6806","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6807","nonce":1198482343}]},"public_addr":"10.20.193.51:6803/1198482343","cluster_addr":"10.20.193.51:6805/1198482343","heartbeat_back_addr":"10.20.193.51:6809/1198482343","heartbeat_front_addr":"10.20.193.51:6807/1198482343","state":["exists","up"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6825","nonce":2540990948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6827","nonce":2540990948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6831","nonce":2540990948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6828","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6829","nonce":2540990948}]},"public_addr":"10.20.193.68:6825/2540990948","cluster_addr":"10.20.193.68:6827/2540990948","heartbeat_back_addr":"10.20.193.68:6831/2540990948","heartbeat_front_addr":"10.20.193.68:6829/2540990948","state":["exists","up"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6808","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6809","nonce":1963822926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6810","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6811","nonce":1963822926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6814","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6815","nonce":1963822926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6812","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6813","nonce":1963822926}]},"public_addr":"10.20.193.125:6809/1963822926","cluster_addr":"10.20.193.125:6811/1963822926","heartbeat_back_addr":"10.20.193.125:6815/1963822926","heartbeat_front_addr":"10.20.193.125:6813/1963822926","state":["exists","up"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6824","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6825","nonce":3567184175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6826","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6827","nonce":3567184175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6830","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6831","nonce":3567184175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6828","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6829","nonce":3567184175}]},"public_addr":"10.20.193.140:6825/3567184175","cluster_addr":"10.20.193.140:6827/3567184175","heartbeat_back_addr":"10.20.193.140:6831/3567184175","heartbeat_front_addr":"10.20.193.140:6829/3567184175","state":["exists","up"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6810","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6811","nonce":1021106712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6812","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6813","nonce":1021106712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6816","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6817","nonce":1021106712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6814","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6815","nonce":1021106712}]},"public_addr":"10.20.193.51:6811/1021106712","cluster_addr":"10.20.193.51:6813/1021106712","heartbeat_back_addr":"10.20.193.51:6817/1021106712","heartbeat_front_addr":"10.20.193.51:6815/1021106712","state":["exists","up"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6817","nonce":3736303751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6819","nonce":3736303751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6823","nonce":3736303751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6821","nonce":3736303751}]},"public_addr":"10.20.193.68:6817/3736303751","cluster_addr":"10.20.193.68:6819/3736303751","heartbeat_back_addr":"10.20.193.68:6823/3736303751","heartbeat_front_addr":"10.20.193.68:6821/3736303751","state":["exists","up"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6824","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6825","nonce":1184344866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6826","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6827","nonce":1184344866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6830","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6831","nonce":1184344866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6828","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6829","nonce":1184344866}]},"public_addr":"10.20.193.125:6825/1184344866","cluster_addr":"10.20.193.125:6827/1184344866","heartbeat_back_addr":"10.20.193.125:6831/1184344866","heartbeat_front_addr":"10.20.193.125:6829/1184344866","state":["exists","up"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6808","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6809","nonce":1140578699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6810","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6811","nonce":1140578699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6814","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6815","nonce":1140578699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6812","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6813","nonce":1140578699}]},"public_addr":"10.20.193.140:6809/1140578699","cluster_addr":"10.20.193.140:6811/1140578699","heartbeat_back_addr":"10.20.193.140:6815/1140578699","heartbeat_front_addr":"10.20.193.140:6813/1140578699","state":["exists","up"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6826","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6827","nonce":142198174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6828","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6829","nonce":142198174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6832","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6833","nonce":142198174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6830","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6831","nonce":142198174}]},"public_addr":"10.20.193.51:6827/142198174","cluster_addr":"10.20.193.51:6829/142198174","heartbeat_back_addr":"10.20.193.51:6833/142198174","heartbeat_front_addr":"10.20.193.51:6831/142198174","state":["exists","up"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6809","nonce":1841964582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6811","nonce":1841964582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6815","nonce":1841964582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6813","nonce":1841964582}]},"public_addr":"10.20.193.68:6809/1841964582","cluster_addr":"10.20.193.68:6811/1841964582","heartbeat_back_addr":"10.20.193.68:6815/1841964582","heartbeat_front_addr":"10.20.193.68:6813/1841964582","state":["exists","up"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6800","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6801","nonce":1951583296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6802","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6803","nonce":1951583296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6806","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6807","nonce":1951583296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6804","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6805","nonce":1951583296}]},"public_addr":"10.20.193.125:6801/1951583296","cluster_addr":"10.20.193.125:6803/1951583296","heartbeat_back_addr":"10.20.193.125:6807/1951583296","heartbeat_front_addr":"10.20.193.125:6805/1951583296","state":["exists","up"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6816","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6817","nonce":2298497951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6818","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6819","nonce":2298497951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6822","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6823","nonce":2298497951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6820","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6821","nonce":2298497951}]},"public_addr":"10.20.193.140:6817/2298497951","cluster_addr":"10.20.193.140:6819/2298497951","heartbeat_back_addr":"10.20.193.140:6823/2298497951","heartbeat_front_addr":"10.20.193.140:6821/2298497951","state":["exists","up"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6818","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6819","nonce":1676524147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6820","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6821","nonce":1676524147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6824","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6825","nonce":1676524147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6822","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6823","nonce":1676524147}]},"public_addr":"10.20.193.51:6819/1676524147","cluster_addr":"10.20.193.51:6821/1676524147","heartbeat_back_addr":"10.20.193.51:6825/1676524147","heartbeat_front_addr":"10.20.193.51:6823/1676524147","state":["exists","up"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6801","nonce":762840177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6803","nonce":762840177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6807","nonce":762840177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6805","nonce":762840177}]},"public_addr":"10.20.193.68:6801/762840177","cluster_addr":"10.20.193.68:6803/762840177","heartbeat_back_addr":"10.20.193.68:6807/762840177","heartbeat_front_addr":"10.20.193.68:6805/762840177","state":["exists","up"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6816","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6817","nonce":498199928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6818","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6819","nonce":498199928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6822","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6823","nonce":498199928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6820","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6821","nonce":498199928}]},"public_addr":"10.20.193.125:6817/498199928","cluster_addr":"10.20.193.125:6819/498199928","heartbeat_back_addr":"10.20.193.125:6823/498199928","heartbeat_front_addr":"10.20.193.125:6821/498199928","state":["exists","up"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6800","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6801","nonce":2657947218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6802","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6803","nonce":2657947218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6806","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6807","nonce":2657947218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6804","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6805","nonce":2657947218}]},"public_addr":"10.20.193.140:6801/2657947218","cluster_addr":"10.20.193.140:6803/2657947218","heartbeat_back_addr":"10.20.193.140:6807/2657947218","heartbeat_front_addr":"10.20.193.140:6805/2657947218","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.521455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.592098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.942138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.663559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.647770+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.811906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.107957+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.927867+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.306670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.794057+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.699761+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.177807+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:24.554387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:30.451 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 10 librados: watch_flush enter 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 10 librados: watch_flush exit 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 20 client.4369.objecter shutdown clearing up homeless session... 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 10 client.4369.objecter successfully canceled tick 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 msgr2=0x7f0c4407b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 0x7f0c4407b230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c6002f450 tx=0x7f0c60033000 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 10 monclient: shutdown 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 msgr2=0x7f0c741e98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 0x7f0c741e98a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c4d0 tx=0x7f0c6c00c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 shutdown_connections 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c44078d70 0x7f0c4407b230 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c7415f860 0x7f0c74153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c7415f480 0x7f0c741538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 --2- 10.20.193.51:0/4236544937 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c74151ff0 0x7f0c741e98a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 >> 10.20.193.51:0/4236544937 conn(0x7f0c7406c890 msgr2=0x7f0c7415e030 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 shutdown_connections 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 -- 10.20.193.51:0/4236544937 wait complete. 2026-02-22T21:37:30.452 INFO:teuthology.misc.health.trial051.stderr:2026-02-22T21:37:30.449+0000 7f0c7c888640 1 librados: shutdown 2026-02-22T21:37:30.461 DEBUG:teuthology.misc:16 of 16 OSDs are up 2026-02-22T21:37:30.462 INFO:tasks.ceph:Creating RBD pool 2026-02-22T21:37:30.462 DEBUG:teuthology.orchestra.run.trial051:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-22T21:37:30.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 10 monclient: get_monmap_and_config 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 10 monclient: build_initial_monmap 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 10 monclient: monmap: 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:30.526995+0000 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:30.526995+0000 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.528 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:30.529 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 1 Processor -- start 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 1 -- start start 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.525+0000 7f0c9adf3640 10 monclient: init 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient: _add_conns ranks=[2,3,4,0,5,1] 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c94156010 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0c940caca0 0x7f0c940c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c940caca0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:30.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f0c940c9020 0x7f0c94073f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-b con 0x7f0c940c9020 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c940c9920 con 0x7f0c94156010 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f0c940cc770 con 0x7f0c940caca0 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f0c940c96d0 con 0x7f0c940c9020 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9adf3640 10 monclient(hunting): authenticate will time out at 1126.107335s 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c99df1640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f0c94156010 0x7f0c940cc060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:44350/0 (socket says 10.20.193.51:44350) 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c99df1640 1 -- 10.20.193.51:0/3281034368 learned_addr learned my addr 10.20.193.51:0/3281034368 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.526+0000 7f0c9a5f2640 1 --2- 10.20.193.51:0/3281034368 >> v2:10.20.193.68:3300/0 conn(0x7f0c940c9020 0x7f0c94073f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): get_auth_request con 0x7f0c940c9020 auth_method 0 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.527+0000 7f0c9a5f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 1 -- 10.20.193.51:0/3281034368 >> v2:10.20.193.51:3300/0 conn(0x7f0c940caca0 msgr2=0x7f0c940c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 1 --2- 10.20.193.51:0/3281034368 >> v2:10.20.193.51:3300/0 conn(0x7f0c940caca0 0x7f0c940c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 1 -- 10.20.193.51:0/3281034368 >> v1:10.20.193.125:6789/0 conn(0x7f0c94156010 legacy=0x7f0c940cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 10 monclient: _finish_hunting 0 2026-02-22T21:37:30.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 1 monclient: found mon.noname-b 2026-02-22T21:37:30.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:30.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.528+0000 7f0c9a5f2640 1 -- 10.20.193.51:0/3281034368 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c940c9c00 con 0x7f0c940c9020 2026-02-22T21:37:30.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.529+0000 7f0c9a5f2640 1 --2- 10.20.193.51:0/3281034368 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94073f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c9406ea50 tx=0x7f0c9002f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d6e5dbd1c0ce55 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 1 -- 10.20.193.51:0/3281034368 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9003c070 con 0x7f0c940c9020 2026-02-22T21:37:30.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:30.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 10 monclient: dump: 2026-02-22T21:37:30.541 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 10 monclient: _finish_auth 0 2026-02-22T21:37:30.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 10 monclient: _check_auth_tickets 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c98def640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c9adf3640 5 monclient: authenticate success, global_id 4372 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.530+0000 7f0c9adf3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 1 -- 10.20.193.51:0/3281034368 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c90037440 con 0x7f0c940c9020 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 1 -- 10.20.193.51:0/3281034368 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90037810 con 0x7f0c940c9020 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c98def640 10 monclient: dump: 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.543 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 4 set_mon_vals no callback set 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 10 monclient: shutdown 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:30.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- 10.20.193.51:0/3281034368 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 msgr2=0x7f0c94073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/3281034368 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94073f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c9406ea50 tx=0x7f0c9002f1c0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- 10.20.193.51:0/3281034368 shutdown_connections 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/3281034368 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94073f30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/3281034368 >> v2:10.20.193.51:3300/0 conn(0x7f0c940caca0 0x7f0c940c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- 10.20.193.51:0/3281034368 >> 10.20.193.51:0/3281034368 conn(0x7f0c940ce340 msgr2=0x7f0c940ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- 10.20.193.51:0/3281034368 shutdown_connections 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- 10.20.193.51:0/3281034368 wait complete. 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 10 monclient: build_initial_monmap 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 10 monclient: monmap: 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:30.545 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 librados: starting msgr at 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 librados: starting objecter 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 Processor -- start 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 -- start start 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 librados: setting wanted keys 2026-02-22T21:37:30.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 1 librados: calling monclient init 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.531+0000 7f0c9adf3640 10 monclient: init 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-b con 0x7f0c940c9020 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c940caca0 0x7f0c940c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-c con 0x7f0c940caca0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c94156010 0x7f0c9414a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c94156010 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0c940ca1c0 con 0x7f0c94156010 2026-02-22T21:37:30.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f0c940ca810 con 0x7f0c940c9020 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0c940ca4a0 con 0x7f0c940caca0 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c9adf3640 10 monclient(hunting): authenticate will time out at 1126.113552s 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c99df1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.532+0000 7f0c995f0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c940caca0 0x7f0c940c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:58842/0 (socket says 10.20.193.51:58842) 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 1 -- 10.20.193.51:0/1023018181 learned_addr learned my addr 10.20.193.51:0/1023018181 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): get_auth_request con 0x7f0c940c9020 auth_method 0 2026-02-22T21:37:30.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): get_auth_request con 0x7f0c940caca0 auth_method 0 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c99df1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.533+0000 7f0c995f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:30.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 -- 10.20.193.51:0/1023018181 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c940caca0 msgr2=0x7f0c940c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c940caca0 0x7f0c940c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 -- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c94156010 msgr2=0x7f0c9414a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c94156010 0x7f0c9414a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 10 monclient: _finish_hunting 0 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 monclient: found mon.noname-b 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.534+0000 7f0c99df1640 1 -- 10.20.193.51:0/1023018181 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c940c75c0 con 0x7f0c940c9020 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.535+0000 7f0c99df1640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c4d0 tx=0x7f0c8400c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.535+0000 7f0c827fc640 10 client.?.objecter ms_handle_connect 0x7f0c940c9020 2026-02-22T21:37:30.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.535+0000 7f0c827fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84017070 con 0x7f0c940c9020 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: dump: 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.551 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84005de0 con 0x7f0c940c9020 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84007290 con 0x7f0c940c9020 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 5 monclient: authenticate success, global_id 4375 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _renew_subs 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c83fff640 4 set_mon_vals no callback set 2026-02-22T21:37:30.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c940c5d00 con 0x7f0c940c9020 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c827fc640 10 monclient: dump: 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:30.553 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _renew_subs 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c940c62c0 con 0x7f0c940c9020 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 1 librados: init done 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.537+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c940c9420 con 0x7f0c940c9020 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.550+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f0c84007430 con 0x7f0c940c9020 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.551+0000 7f0c827fc640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 0x7f0c6407b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.551+0000 7f0c995f0640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 0x7f0c6407b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.551+0000 7f0c995f0640 10 monclient: get_auth_request con 0x7f0c64078da0 auth_method 0 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.551+0000 7f0c827fc640 10 client.4375.objecter ms_handle_connect 0x7f0c64078da0 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.551+0000 7f0c995f0640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 0x7f0c6407b260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c9414eac0 tx=0x7f0c88009210 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.553+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 8563+0+0 (secure 0 0 0) 0x7f0c84007750 con 0x7f0c940c9020 2026-02-22T21:37:30.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.553+0000 7f0c827fc640 10 client.4375.objecter ms_dispatch2 0x7f0c94000b90 osd_map(24..24 src has 1..24) 2026-02-22T21:37:30.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.553+0000 7f0c827fc640 3 client.4375.objecter handle_osd_map got epochs [24,24] > 0 2026-02-22T21:37:30.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.553+0000 7f0c827fc640 3 client.4375.objecter handle_osd_map decoding full epoch 24 2026-02-22T21:37:30.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.553+0000 7f0c827fc640 20 client.4375.objecter dump_active .. 0 homeless 2026-02-22T21:37:30.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.653+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0c84021020 con 0x7f0c940c9020 2026-02-22T21:37:30.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.653+0000 7f0c827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:30.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.653+0000 7f0c827fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:30.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.737+0000 7f0c9adf3640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2026-02-22T21:37:30.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.737+0000 7f0c9adf3640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2026-02-22T21:37:30.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.737+0000 7f0c9adf3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:30.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:30.737+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f0c940cb680 con 0x7f0c940c9020 2026-02-22T21:37:31.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.755+0000 7f0c827fc640 1 -- 10.20.193.51:0/1023018181 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v26) ==== 109+0+0 (secure 0 0 0) 0x7f0c84061330 con 0x7f0c940c9020 2026-02-22T21:37:31.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.755+0000 7f0c827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2026-02-22T21:37:31.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.755+0000 7f0c827fc640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2026-02-22T21:37:31.758 INFO:teuthology.orchestra.run.trial051.stderr:pool 'rbd' created 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 10 librados: watch_flush enter 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 10 librados: watch_flush exit 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 20 client.4375.objecter shutdown clearing up homeless session... 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 10 client.4375.objecter successfully canceled tick 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 msgr2=0x7f0c6407b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 0x7f0c6407b260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c9414eac0 tx=0x7f0c88009210 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 10 monclient: shutdown 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 msgr2=0x7f0c94151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c4d0 tx=0x7f0c8400c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 shutdown_connections 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0c64078da0 0x7f0c6407b260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0c94156010 0x7f0c9414a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0c940caca0 0x7f0c940c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 --2- 10.20.193.51:0/1023018181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0c940c9020 0x7f0c94151a50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 >> 10.20.193.51:0/1023018181 conn(0x7f0c940ce340 msgr2=0x7f0c9415da70 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:31.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 shutdown_connections 2026-02-22T21:37:31.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 -- 10.20.193.51:0/1023018181 wait complete. 2026-02-22T21:37:31.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.757+0000 7f0c9adf3640 1 librados: shutdown 2026-02-22T21:37:31.768 DEBUG:teuthology.orchestra.run.trial051:> rbd --cluster ceph pool init rbd 2026-02-22T21:37:31.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.796+0000 7f661cb5e0c0 10 monclient: get_monmap_and_config 2026-02-22T21:37:31.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.796+0000 7f661cb5e0c0 10 monclient: build_initial_monmap 2026-02-22T21:37:31.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.796+0000 7f661cb5e0c0 10 monclient: monmap: 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:31.798504+0000 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:31.798504+0000 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:31.800 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:31.801 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:31.801 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.797+0000 7f661cb5e0c0 1 Processor -- start 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.797+0000 7f661cb5e0c0 1 -- start start 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.797+0000 7f661cb5e0c0 10 monclient: init 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient: _reopen_session rank -1 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient: _add_conns ranks=[0,3,1,4,5,2] 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-a-legacy con 0x5596f4af9760 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 1 --2- >> v2:10.20.193.51:3300/0 conn(0x5596f4a6aa90 0x5596f4a6c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:31.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-a con 0x5596f4a6aa90 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x5596f4a2e990 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5596f4891d20 con 0x5596f4af9760 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5596f4a2c330 con 0x5596f4a2e990 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x5596f48be720 con 0x5596f4a6aa90 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): _renew_subs 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661cb5e0c0 10 monclient(hunting): authenticate will time out at 1127.379271s 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661ae23640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x5596f4a6aa90 0x5596f4a6c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661c8ad640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x5596f4a2e990 0x5596f4a2db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:59044/0 (socket says 10.20.193.51:59044) 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661c8ad640 1 -- 10.20.193.51:0/1157077862 learned_addr learned my addr 10.20.193.51:0/1157077862 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:31.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661ae23640 10 monclient(hunting): get_auth_request con 0x5596f4a6aa90 auth_method 0 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661ae23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661ae23640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.798+0000 7f661ae23640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.799+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.799+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.799+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 1 -- 10.20.193.51:0/1157077862 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3010343474 0 0) 0x5596f4a2c330 con 0x5596f4a2e990 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 10 monclient(hunting): _negotiate 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 10 monclient(hunting): my global_id is 4384 2026-02-22T21:37:31.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.800+0000 7f661a622640 1 -- 10.20.193.51:0/1157077862 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6608001e10 con 0x5596f4a2e990 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 1 -- 10.20.193.51:0/1157077862 >> v1:10.20.193.68:6789/0 conn(0x5596f4a2e990 legacy=0x5596f4a2db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 1 -- 10.20.193.51:0/1157077862 >> v1:10.20.193.51:6789/0 conn(0x5596f4af9760 legacy=0x5596f4a6a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 10 monclient: _finish_hunting 0 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 1 monclient: found mon.noname-a 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:31.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.801+0000 7f661ae23640 1 -- 10.20.193.51:0/1157077862 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f4a695c0 con 0x5596f4a6aa90 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.803+0000 7f661ae23640 1 --2- 10.20.193.51:0/1157077862 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4a6aa90 0x5596f4a6c120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f660400a520 tx=0x7f66040586c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a557674b854349d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 1 -- 10.20.193.51:0/1157077862 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660406d070 con 0x5596f4a6aa90 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: dump: 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.813 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: _finish_auth 0 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: _check_auth_tickets 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 1 -- 10.20.193.51:0/1157077862 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6604060d80 con 0x5596f4a6aa90 2026-02-22T21:37:31.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 1 -- 10.20.193.51:0/1157077862 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604068680 con 0x5596f4a6aa90 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 5 monclient: authenticate success, global_id 4377 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 10 monclient: get_monmap_and_config success 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 4 set_mon_vals no callback set 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661a622640 10 monclient: dump: 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.815 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 10 monclient: shutdown 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1157077862 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4a6aa90 msgr2=0x5596f4a6c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1157077862 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4a6aa90 0x5596f4a6c120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f660400a520 tx=0x7f66040586c0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1157077862 shutdown_connections 2026-02-22T21:37:31.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1157077862 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4a6aa90 0x5596f4a6c120 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1157077862 >> 10.20.193.51:0/1157077862 conn(0x5596f4a6ff90 msgr2=0x5596f4a703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1157077862 shutdown_connections 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.805+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1157077862 wait complete. 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient: build_initial_monmap 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient: monmap: 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:31.817 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 librados: starting msgr at 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 librados: starting objecter 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 Processor -- start 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 -- start start 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 librados: setting wanted keys 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 librados: calling monclient init 2026-02-22T21:37:31.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient: init 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient: _reopen_session rank -1 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x5596f4a2e990 0x5596f4af5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.806+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-c con 0x5596f4a2e990 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x5596f4a6aa90 0x5596f4b0f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-b con 0x5596f4a6aa90 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 0x5596f4b0f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): picked mon.noname-a con 0x5596f4af9760 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x5596f490edb0 con 0x5596f4af9760 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:31.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x5596f4912e80 con 0x5596f4a6aa90 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x5596f48ade50 con 0x5596f4a2e990 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): _renew_subs 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661cb5e0c0 10 monclient(hunting): authenticate will time out at 1127.388026s 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 0x5596f4b0f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 0x5596f4b0f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:55284/0 (socket says 10.20.193.51:55284) 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 learned_addr learned my addr 10.20.193.51:0/1929258523 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): get_auth_request con 0x5596f4af9760 auth_method 0 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:31.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x5596f4a2e990 0x5596f4af5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x5596f4a6aa90 0x5596f4b0f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661c8ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): get_auth_request con 0x5596f4a6aa90 auth_method 0 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): get_auth_request con 0x5596f4a2e990 auth_method 0 2026-02-22T21:37:31.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661ae23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.807+0000 7f661b624640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661b624640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x5596f4a2e990 msgr2=0x5596f4af5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x5596f4a2e990 0x5596f4af5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x5596f4a6aa90 msgr2=0x5596f4b0f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x5596f4a6aa90 0x5596f4b0f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 10 monclient: _finish_hunting 0 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 monclient: found mon.noname-a 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.808+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f4b124e0 con 0x5596f4af9760 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.809+0000 7f661c8ad640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 0x5596f4b0f7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f660c006fc0 tx=0x7f660c00b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.809+0000 7f66037fe640 10 client.?.objecter ms_handle_connect 0x5596f4af9760 2026-02-22T21:37:31.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.809+0000 7f66037fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:31.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c016070 con 0x5596f4af9760 2026-02-22T21:37:31.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:31.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:31.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: dump: 2026-02-22T21:37:31.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:31.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c0062d0 con 0x5596f4af9760 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c004740 con 0x5596f4af9760 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 5 monclient: authenticate success, global_id 4383 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 monclient: _renew_subs 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5596f4b127c0 con 0x5596f4af9760 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f6619e21640 4 set_mon_vals no callback set 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f66037fe640 10 monclient: dump: 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:31.828 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 monclient: _renew_subs 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f4b12d80 con 0x5596f4af9760 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 1 librados: init done 2026-02-22T21:37:31.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.811+0000 7f661cb5e0c0 10 librados: wait_for_osdmap waiting 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.823+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 4) ==== 99845+0+0 (secure 0 0 0) 0x7f660c0048e0 con 0x5596f4af9760 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.824+0000 7f66037fe640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65fc076e20 0x7f65fc0792e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.824+0000 7f661b624640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65fc076e20 0x7f65fc0792e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.824+0000 7f661b624640 10 monclient: get_auth_request con 0x7f65fc076e20 auth_method 0 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.824+0000 7f661b624640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65fc076e20 0x7f65fc0792e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6610000c00 tx=0x7f6610001790 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:31.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.824+0000 7f66037fe640 10 client.4383.objecter ms_handle_connect 0x7f65fc076e20 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 8905+0+0 (secure 0 0 0) 0x7f660c05bbf0 con 0x5596f4af9760 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_map(26..26 src has 1..26) 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f66037fe640 3 client.4383.objecter handle_osd_map got epochs [26,26] > 0 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f66037fe640 3 client.4383.objecter handle_osd_map decoding full epoch 26 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f66037fe640 20 client.4383.objecter dump_active .. 0 homeless 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f661cb5e0c0 10 librados: wait_for_osdmap done waiting 2026-02-22T21:37:31.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f661cb5e0c0 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2026-02-22T21:37:31.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f661cb5e0c0 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2026-02-22T21:37:31.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f661cb5e0c0 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:31.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:31.825+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x5596f4af57d0 con 0x5596f4af9760 2026-02-22T21:37:32.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.496+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f660c004c00 con 0x5596f4af9760 2026-02-22T21:37:32.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.794+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v27) ==== 141+0+0 (secure 0 0 0) 0x7f660c087e60 con 0x5596f4af9760 2026-02-22T21:37:32.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.794+0000 7f66037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2026-02-22T21:37:32.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.794+0000 7f66037fe640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2026-02-22T21:37:32.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.794+0000 7f661cb5e0c0 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:32.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.794+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6604060d80 con 0x5596f4af9760 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 8 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f660c087530 con 0x5596f4af9760 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f66037fe640 10 monclient: handle_get_version_reply finishing 1 version 27 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f6619e21640 10 client.4383.objecter _get_latest_version latest 27, waiting 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f6619e21640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f6619e21640 10 monclient: _renew_subs 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f6619e21640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:32.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.832+0000 7f6619e21640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=27}) -- 0x7f65f4003c90 con 0x5596f4af9760 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 9 ==== osd_map(27..27 src has 1..27) ==== 674+0+0 (secure 0 0 0) 0x7f660c072410 con 0x5596f4af9760 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_map(27..27 src has 1..27) 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f66037fe640 3 client.4383.objecter handle_osd_map got epochs [27,27] > 26 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f66037fe640 3 client.4383.objecter handle_osd_map decoding incremental epoch 27 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f66037fe640 20 client.4383.objecter dump_active .. 0 homeless 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 10 librados: create oid=rbd_trash nspace= 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit op 0x5596f4a70980 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target epoch 27 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 10 client.4383.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [14,5] primary 14 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x5596f4ae5fa0 0x5596f4aeb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 20 client.4383.objecter _get_session s=0x5596f48ae230 osd=14 3 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.14 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 20 client.4383.objecter get_session s=0x5596f48ae230 osd=14 3 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 15 client.4383.objecter _session_op_assign 14 1 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 15 client.4383.objecter _send_op 1 to 2.3 on osd.14 2026-02-22T21:37:32.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5596f4ae63a0 con 0x5596f4ae5fa0 2026-02-22T21:37:32.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 20 client.4383.objecter put_session s=0x5596f48ae230 osd=14 4 2026-02-22T21:37:32.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.834+0000 7f661cb5e0c0 5 client.4383.objecter 1 in flight 2026-02-22T21:37:32.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.837+0000 7f661ae23640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x5596f4ae5fa0 0x5596f4aeb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:32.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.838+0000 7f661ae23640 10 monclient: get_auth_request con 0x5596f4ae5fa0 auth_method 0 2026-02-22T21:37:32.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.909+0000 7f661ae23640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x5596f4ae5fa0 0x5596f4aeb090 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.14 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:32.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:32.909+0000 7f66037fe640 10 client.4383.objecter ms_handle_connect 0x5596f4ae5fa0 2026-02-22T21:37:33.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 1 -- 10.20.193.51:0/1929258523 <== osd.14 v2:10.20.193.125:6816/498199928 1 ==== osd_op_reply(1 rbd_trash [create] v27'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f6604070040 con 0x5596f4ae5fa0 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_op_reply(1 rbd_trash [create] v27'1 uv1 ondisk = 0) 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 10 client.4383.objecter in handle_osd_op_reply 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 7 client.4383.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 10 client.4383.objecter op 0 rval 0 len 0 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 15 client.4383.objecter handle_osd_op_reply completed tid 1 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 15 client.4383.objecter _finish_op 1 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 20 client.4383.objecter put_session s=0x5596f48ae230 osd=14 4 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 15 client.4383.objecter _session_op_remove 14 1 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661ae23640 5 client.4383.objecter 0 in flight 2026-02-22T21:37:33.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.774+0000 7f661cb5e0c0 10 librados: Objecter returned from create r=0 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit op 0x5596f4b6ba40 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target epoch 27 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 10 client.4383.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [13,7] primary 13 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x5596f4b6c5f0 0x5596f4b6ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 20 client.4383.objecter _get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.13 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 20 client.4383.objecter get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 15 client.4383.objecter _session_op_assign 13 2 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 15 client.4383.objecter _send_op 2 to 2.2 on osd.13 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5596f4b6f000 con 0x5596f4b6c5f0 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661cb5e0c0 5 client.4383.objecter 1 in flight 2026-02-22T21:37:33.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661b624640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x5596f4b6c5f0 0x5596f4b6ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:33.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.775+0000 7f661b624640 10 monclient: get_auth_request con 0x5596f4b6c5f0 auth_method 0 2026-02-22T21:37:33.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.778+0000 7f661b624640 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x5596f4b6c5f0 0x5596f4b6ea80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.13 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:33.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.778+0000 7f66037fe640 10 client.4383.objecter ms_handle_connect 0x5596f4b6c5f0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 1 -- 10.20.193.51:0/1929258523 <== osd.13 v2:10.20.193.68:6800/762840177 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f6610002050 con 0x5596f4b6c5f0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 10 client.4383.objecter in handle_osd_op_reply 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 7 client.4383.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 10 client.4383.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 10 client.4383.objecter op 0 rval 0 len 0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 15 client.4383.objecter handle_osd_op_reply completed tid 2 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 15 client.4383.objecter _finish_op 2 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 15 client.4383.objecter _session_op_remove 13 2 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661b624640 5 client.4383.objecter 0 in flight 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit op 0x5596f4b6ac20 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target epoch 27 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 20 client.4383.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 10 client.4383.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [13,7] primary 13 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 20 client.4383.objecter _get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 10 client.4383.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.13 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 20 client.4383.objecter get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 15 client.4383.objecter _session_op_assign 13 3 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 15 client.4383.objecter _send_op 3 to 2.2 on osd.13 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5596f4b6b0d0 con 0x5596f4b6c5f0 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:33.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.833+0000 7f661cb5e0c0 5 client.4383.objecter 1 in flight 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 1 -- 10.20.193.51:0/1929258523 <== osd.13 v2:10.20.193.68:6800/762840177 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f6610002050 con 0x5596f4b6c5f0 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 10 client.4383.objecter in handle_osd_op_reply 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 7 client.4383.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 10 client.4383.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 10 client.4383.objecter op 0 rval 0 len 0 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 15 client.4383.objecter handle_osd_op_reply completed tid 3 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 15 client.4383.objecter _finish_op 3 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 15 client.4383.objecter _session_op_remove 13 3 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f661b624640 5 client.4383.objecter 0 in flight 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f6619620640 10 client.4383.objecter allocate_selfmanaged_snap; pool: 2 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f6619620640 10 client.4383.objecter pool_op_submit 4 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f6619620640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:33.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:33.838+0000 7f6619620640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f65f80030d0 con 0x5596f4af9760 2026-02-22T21:37:34.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.807+0000 7f66027fc640 10 monclient: tick 2026-02-22T21:37:34.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.807+0000 7f66027fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:34.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.807+0000 7f66027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:34.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.807+0000 7f66027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 10 ==== pool_op_reply(tid 4 (0) Success v29) ==== 55+0+0 (secure 0 0 0) 0x7f660c087760 con 0x5596f4af9760 2026-02-22T21:37:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 pool_op_reply(tid 4 (0) Success v29) 2026-02-22T21:37:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 client.4383.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v29) 2026-02-22T21:37:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 client.4383.objecter have request 4 at 0x7f65f8003060 Op: create unmanaged snap 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 20 client.4383.objecter waiting for client to reach epoch 29 before calling back 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 monclient: _renew_subs 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=28}) -- 0x7f65fc07ef10 con 0x5596f4af9760 2026-02-22T21:37:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.880+0000 7f66037fe640 10 client.4383.objecter done 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.883+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 11 ==== osd_map(28..29 src has 1..29) ==== 1216+0+0 (secure 0 0 0) 0x7f660c098020 con 0x5596f4af9760 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_map(28..29 src has 1..29) 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f66037fe640 3 client.4383.objecter handle_osd_map got epochs [28,29] > 27 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f66037fe640 3 client.4383.objecter handle_osd_map decoding incremental epoch 28 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f66037fe640 3 client.4383.objecter handle_osd_map decoding incremental epoch 29 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f66037fe640 20 client.4383.objecter dump_active .. 0 homeless 2026-02-22T21:37:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 librados: queue_aio_write 0x5596f4b66fd0 completion 0x5596f48d25f0 write_seq 1 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 10 client.4383.objecter _op_submit op 0x7f65f4006380 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 client.4383.objecter _calc_target epoch 29 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 client.4383.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 10 client.4383.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [13,7] primary 13 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 client.4383.objecter _get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 10 client.4383.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.13 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 client.4383.objecter get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 15 client.4383.objecter _session_op_assign 13 5 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 15 client.4383.objecter _send_op 5 to 2.2 on osd.13 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x7f65f4006a20 con 0x5596f4b6c5f0 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:34.884+0000 7f6619e21640 5 client.4383.objecter 1 in flight 2026-02-22T21:37:35.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 1 -- 10.20.193.51:0/1929258523 <== osd.13 v2:10.20.193.68:6800/762840177 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v29'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f6610002050 con 0x5596f4b6c5f0 2026-02-22T21:37:35.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_op_reply(5 rbd_info [create,write 0~8] v29'1 uv1 ondisk = 0) 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 10 client.4383.objecter in handle_osd_op_reply 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 7 client.4383.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 10 client.4383.objecter op 0 rval 0 len 0 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 10 client.4383.objecter op 1 rval 0 len 0 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 15 client.4383.objecter handle_osd_op_reply completed tid 5 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 15 client.4383.objecter _finish_op 5 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 15 client.4383.objecter _session_op_remove 13 5 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 5 client.4383.objecter 0 in flight 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f661b624640 20 librados: complete_aio_write 0x5596f48d25f0 2026-02-22T21:37:35.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f6619620640 10 client.4383.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2026-02-22T21:37:35.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f6619620640 10 client.4383.objecter pool_op_submit 6 2026-02-22T21:37:35.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f6619620640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:35.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.133+0000 7f6619620640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v29) -- 0x7f65f8004520 con 0x5596f4af9760 2026-02-22T21:37:35.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.807+0000 7f66027fc640 10 monclient: tick 2026-02-22T21:37:35.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.807+0000 7f66027fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:35.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:35.807+0000 7f66027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 12 ==== pool_op_reply(tid 6 (0) Success v30) ==== 43+0+0 (secure 0 0 0) 0x7f660c08bb10 con 0x5596f4af9760 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 pool_op_reply(tid 6 (0) Success v30) 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 client.4383.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v30) 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 client.4383.objecter have request 6 at 0x7f65f8003060 Op: delete unmanaged snap 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 20 client.4383.objecter waiting for client to reach epoch 30 before calling back 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 monclient: _renew_subs 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:36.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=30}) -- 0x7f65fc07ef10 con 0x5596f4af9760 2026-02-22T21:37:36.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.763+0000 7f66037fe640 10 client.4383.objecter done 2026-02-22T21:37:36.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f66037fe640 1 -- 10.20.193.51:0/1929258523 <== mon.0 v2:10.20.193.51:3300/0 13 ==== osd_map(30..30 src has 1..30) ==== 654+0+0 (secure 0 0 0) 0x7f660c08b030 con 0x5596f4af9760 2026-02-22T21:37:36.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f66037fe640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_map(30..30 src has 1..30) 2026-02-22T21:37:36.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f66037fe640 3 client.4383.objecter handle_osd_map got epochs [30,30] > 29 2026-02-22T21:37:36.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f66037fe640 3 client.4383.objecter handle_osd_map decoding incremental epoch 30 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f66037fe640 20 client.4383.objecter dump_active .. 0 homeless 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 librados: queue_aio_write 0x5596f4b66fd0 completion 0x7f65f8001d60 write_seq 2 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 10 client.4383.objecter _op_submit op 0x7f65f4006380 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 client.4383.objecter _calc_target epoch 30 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 client.4383.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 10 client.4383.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [13,7] primary 13 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 client.4383.objecter _get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 10 client.4383.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.13 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 client.4383.objecter get_session s=0x5596f4b6bf70 osd=13 3 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 15 client.4383.objecter _session_op_assign 13 7 2026-02-22T21:37:36.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 15 client.4383.objecter _send_op 7 to 2.2 on osd.13 2026-02-22T21:37:36.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 1 -- 10.20.193.51:0/1929258523 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) -- 0x7f65f4006dc0 con 0x5596f4b6c5f0 2026-02-22T21:37:36.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:36.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.768+0000 7f6619e21640 5 client.4383.objecter 1 in flight 2026-02-22T21:37:36.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.807+0000 7f66027fc640 10 monclient: tick 2026-02-22T21:37:36.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.807+0000 7f66027fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:36.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.807+0000 7f66027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:36.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.812+0000 7f6603fff640 10 client.4383.objecter tick 2026-02-22T21:37:36.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 1 -- 10.20.193.51:0/1929258523 <== osd.13 v2:10.20.193.68:6800/762840177 4 ==== osd_op_reply(7 rbd_info [write 0~19] v30'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6610002050 con 0x5596f4b6c5f0 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 10 client.4383.objecter ms_dispatch2 0x5596f49299a0 osd_op_reply(7 rbd_info [write 0~19] v30'2 uv2 ondisk = 0) 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 10 client.4383.objecter in handle_osd_op_reply 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 7 client.4383.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 10 client.4383.objecter op 0 rval 0 len 0 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 15 client.4383.objecter handle_osd_op_reply completed tid 7 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 15 client.4383.objecter _finish_op 7 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 4 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 15 client.4383.objecter _session_op_remove 13 7 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 5 client.4383.objecter 0 in flight 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661b624640 20 librados: complete_aio_write 0x7f65f8001d60 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 librados: watch_flush enter 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 librados: watch_flush exit 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 client.4383.objecter close_session for osd.13 2026-02-22T21:37:36.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x5596f4b6c5f0 msgr2=0x5596f4b6ea80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x5596f4b6c5f0 0x5596f4b6ea80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 20 client.4383.objecter put_session s=0x5596f4b6bf70 osd=13 1 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 client.4383.objecter close_session for osd.14 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x5596f4ae5fa0 msgr2=0x5596f4aeb090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x5596f4ae5fa0 0x5596f4aeb090 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 20 client.4383.objecter put_session s=0x5596f48ae230 osd=14 1 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 20 client.4383.objecter shutdown clearing up homeless session... 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 client.4383.objecter successfully canceled tick 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65fc076e20 msgr2=0x7f65fc0792e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f65fc076e20 0x7f65fc0792e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6610000c00 tx=0x7f6610001790 comp rx=0 tx=0).stop 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 10 monclient: shutdown 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 msgr2=0x5596f4b0f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 --2- 10.20.193.51:0/1929258523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x5596f4af9760 0x5596f4b0f7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f660c006fc0 tx=0x7f660c00b580 comp rx=0 tx=0).stop 2026-02-22T21:37:36.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 reap_dead start 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661c8ad640 1 -- 10.20.193.51:0/1929258523 reap_dead start 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 shutdown_connections 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 >> 10.20.193.51:0/1929258523 conn(0x5596f4a6ff90 msgr2=0x5596f4af39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 shutdown_connections 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 -- 10.20.193.51:0/1929258523 wait complete. 2026-02-22T21:37:36.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.858+0000 7f661cb5e0c0 1 librados: shutdown 2026-02-22T21:37:36.864 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-22T21:37:36.864 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-22T21:37:36.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-22T21:37:36.865 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-22T21:37:36.865 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 10 monclient: get_monmap_and_config 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 10 monclient: build_initial_monmap 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 10 monclient: monmap: 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:36.974 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:36.972914+0000 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:36.972914+0000 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 1 Processor -- start 2026-02-22T21:37:36.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 1 -- start start 2026-02-22T21:37:36.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.972+0000 7f3229dfe640 10 monclient: init 2026-02-22T21:37:36.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:36.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient: _add_conns ranks=[5,4,1,2,0,3] 2026-02-22T21:37:36.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f3224150220 0x7f32240c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:36.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-c con 0x7f3224150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:36.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3224150600 0x7f322415c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-b con 0x7f3224150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f322415c540 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f322415c9b0 con 0x7f322415c540 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f322415a920 con 0x7f3224150600 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f322415af60 con 0x7f3224150220 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3229dfe640 10 monclient(hunting): authenticate will time out at 1132.553234s 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.973+0000 7f3222ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3224150600 0x7f322415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3223fff640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f322415c540 0x7f322406fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:59060/0 (socket says 10.20.193.51:59060) 2026-02-22T21:37:36.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3223fff640 1 -- 10.20.193.51:0/1289074242 learned_addr learned my addr 10.20.193.51:0/1289074242 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 1 --2- 10.20.193.51:0/1289074242 >> v2:10.20.193.125:3300/0 conn(0x7f3224150220 0x7f32240c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): get_auth_request con 0x7f3224150600 auth_method 0 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): get_auth_request con 0x7f3224150220 auth_method 0 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:36.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.974+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 1 -- 10.20.193.51:0/1289074242 >> v2:10.20.193.125:3300/0 conn(0x7f3224150220 msgr2=0x7f32240c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 1 --2- 10.20.193.51:0/1289074242 >> v2:10.20.193.125:3300/0 conn(0x7f3224150220 0x7f32240c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 1 -- 10.20.193.51:0/1289074242 >> v1:10.20.193.68:6789/0 conn(0x7f322415c540 legacy=0x7f322406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 1 monclient: found mon.noname-b 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:36.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.975+0000 7f3222ffd640 1 -- 10.20.193.51:0/1289074242 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322415b240 con 0x7f3224150600 2026-02-22T21:37:36.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.979+0000 7f3222ffd640 1 --2- 10.20.193.51:0/1289074242 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150600 0x7f322415c000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3224070940 tx=0x7f321402eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c72b082817a03059 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:36.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.981+0000 7f32227fc640 1 -- 10.20.193.51:0/1289074242 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321403c070 con 0x7f3224150600 2026-02-22T21:37:36.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.981+0000 7f32227fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:36.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.981+0000 7f32227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:36.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.981+0000 7f32227fc640 10 monclient: dump: 2026-02-22T21:37:36.987 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:36.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 1 -- 10.20.193.51:0/1289074242 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321402fb80 con 0x7f3224150600 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 5 monclient: authenticate success, global_id 4393 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 1 -- 10.20.193.51:0/1289074242 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214037440 con 0x7f3224150600 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 4 set_mon_vals no callback set 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f32227fc640 10 monclient: dump: 2026-02-22T21:37:36.989 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 10 monclient: shutdown 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:36.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- 10.20.193.51:0/1289074242 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150600 msgr2=0x7f322415c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 --2- 10.20.193.51:0/1289074242 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150600 0x7f322415c000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3224070940 tx=0x7f321402eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- 10.20.193.51:0/1289074242 shutdown_connections 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 --2- 10.20.193.51:0/1289074242 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150600 0x7f322415c000 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 --2- 10.20.193.51:0/1289074242 >> v2:10.20.193.125:3300/0 conn(0x7f3224150220 0x7f32240c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- 10.20.193.51:0/1289074242 >> 10.20.193.51:0/1289074242 conn(0x7f32240cd7b0 msgr2=0x7f32240cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- 10.20.193.51:0/1289074242 shutdown_connections 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- 10.20.193.51:0/1289074242 wait complete. 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 10 monclient: build_initial_monmap 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 10 monclient: monmap: 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:36.991 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 librados: starting msgr at 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 librados: starting objecter 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 Processor -- start 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.982+0000 7f3229dfe640 1 -- start start 2026-02-22T21:37:36.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 librados: setting wanted keys 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 librados: calling monclient init 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient: init 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-b con 0x7f3224150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3224150600 0x7f3224158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-c con 0x7f3224150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f322415c540 0x7f322414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): picked mon.noname-a con 0x7f322415c540 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:36.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f322415bae0 con 0x7f322415c540 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f322415b800 con 0x7f3224150220 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f32240703b0 con 0x7f3224150600 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3229dfe640 10 monclient(hunting): authenticate will time out at 1132.563418s 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3223fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f322415c540 0x7f322414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:58904/0 (socket says 10.20.193.51:58904) 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3222ffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3224150600 0x7f3224158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 1 -- 10.20.193.51:0/3635501419 learned_addr learned my addr 10.20.193.51:0/3635501419 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:36.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 10 monclient(hunting): get_auth_request con 0x7f3224150220 auth_method 0 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f32237fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3223fff640 10 monclient(hunting): get_auth_request con 0x7f322415c540 auth_method 0 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3223fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.983+0000 7f3223fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f32237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): get_auth_request con 0x7f3224150600 auth_method 0 2026-02-22T21:37:36.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3223fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.984+0000 7f3222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 -- 10.20.193.51:0/3635501419 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3224150600 msgr2=0x7f3224158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3224150600 0x7f3224158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 -- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f322415c540 msgr2=0x7f322414a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:36.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f322415c540 0x7f322414a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:36.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:36.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:36.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:36.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:36.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.985+0000 7f32237fe640 1 -- 10.20.193.51:0/3635501419 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3224157720 con 0x7f3224150220 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.986+0000 7f32237fe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f321000ebe0 tx=0x7f321000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.986+0000 7f31fffff640 10 client.?.objecter ms_handle_connect 0x7f3224150220 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.986+0000 7f31fffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210017070 con 0x7f3224150220 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: dump: 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:36.999 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: _finish_auth 0 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 5 monclient: authenticate success, global_id 4399 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 monclient: _renew_subs 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3224157a00 con 0x7f3224150220 2026-02-22T21:37:37.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 monclient: _renew_subs 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3224157fc0 con 0x7f3224150220 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 1 librados: init done 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321000cc40 con 0x7f3224150220 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210021060 con 0x7f3224150220 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f31fffff640 10 monclient: dump: 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.001 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3221ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.988+0000 7f3229dfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.989+0000 7f3229dfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.989+0000 7f3229dfe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:36.989+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f322415c000 con 0x7f3224150220 2026-02-22T21:37:37.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.095+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f3210013470 con 0x7f3224150220 2026-02-22T21:37:37.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.096+0000 7f31fffff640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 0x7f31f407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.096+0000 7f3222ffd640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 0x7f31f407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.096+0000 7f3222ffd640 10 monclient: get_auth_request con 0x7f31f4078e00 auth_method 0 2026-02-22T21:37:37.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.096+0000 7f3222ffd640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 0x7f31f407b2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3214039000 tx=0x7f3214033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.096+0000 7f31fffff640 10 client.4399.objecter ms_handle_connect 0x7f31f4078e00 2026-02-22T21:37:37.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.097+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f321005e820 con 0x7f3224150220 2026-02-22T21:37:37.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.097+0000 7f31fffff640 10 client.4399.objecter ms_dispatch2 0x7f3224000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:37.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.097+0000 7f31fffff640 3 client.4399.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:37.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.097+0000 7f31fffff640 3 client.4399.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:37.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.097+0000 7f31fffff640 20 client.4399.objecter dump_active .. 0 homeless 2026-02-22T21:37:37.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.103+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f32100627e0 con 0x7f3224150220 2026-02-22T21:37:37.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.103+0000 7f31fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.103+0000 7f31fffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:37.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.211+0000 7f3229dfe640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2026-02-22T21:37:37.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.211+0000 7f3229dfe640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2026-02-22T21:37:37.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.211+0000 7f3229dfe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.211+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3224074da0 con 0x7f3224150220 2026-02-22T21:37:37.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.444+0000 7f31fffff640 1 -- 10.20.193.51:0/3635501419 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+191888 (secure 0 0 0) 0x7f3210013e20 con 0x7f3224150220 2026-02-22T21:37:37.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.445+0000 7f31fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2026-02-22T21:37:37.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.445+0000 7f31fffff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:37.448 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 10 librados: watch_flush enter 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 10 librados: watch_flush exit 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 20 client.4399.objecter shutdown clearing up homeless session... 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 10 client.4399.objecter successfully canceled tick 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 msgr2=0x7f31f407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 0x7f31f407b2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3214039000 tx=0x7f3214033000 comp rx=0 tx=0).stop 2026-02-22T21:37:37.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 10 monclient: shutdown 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 msgr2=0x7f32241587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f321000ebe0 tx=0x7f321000c520 comp rx=0 tx=0).stop 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 shutdown_connections 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f31f4078e00 0x7f31f407b2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f322415c540 0x7f322414a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3224150600 0x7f3224158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 --2- 10.20.193.51:0/3635501419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3224150220 0x7f32241587c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 >> 10.20.193.51:0/3635501419 conn(0x7f32240cd7b0 msgr2=0x7f322414cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 shutdown_connections 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 -- 10.20.193.51:0/3635501419 wait complete. 2026-02-22T21:37:37.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.447+0000 7f3229dfe640 1 librados: shutdown 2026-02-22T21:37:37.458 INFO:teuthology.orchestra.run.trial051.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6800","nonce":3302837800},{"type":"v1","addr":"10.20.193.51:6801","nonce":3302837800}]},"active_addr":"10.20.193.51:6801/3302837800","active_change":"2026-02-22T21:35:49.573671+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.51:0","nonce":386363810}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.51:0","nonce":3979521181}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.51:0","nonce":2352749568}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.51:0","nonce":4246901174}]}]} 2026-02-22T21:37:37.464 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-22T21:37:37.464 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-22T21:37:37.465 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:37.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.565+0000 7f41b5de4640 10 monclient: get_monmap_and_config 2026-02-22T21:37:37.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.565+0000 7f41b5de4640 10 monclient: build_initial_monmap 2026-02-22T21:37:37.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.565+0000 7f41b5de4640 10 monclient: monmap: 2026-02-22T21:37:37.567 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:37.567 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:37.566136+0000 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:37.566136+0000 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.565+0000 7f41b5de4640 1 Processor -- start 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.565+0000 7f41b5de4640 1 -- start start 2026-02-22T21:37:37.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient: init 2026-02-22T21:37:37.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:37.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient: _add_conns ranks=[0,4,2,5,1,3] 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41b0152ab0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f41b006d9d0 0x7f41b00c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-b con 0x7f41b006d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41b00c93d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b006f1d0 con 0x7f41b0152ab0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b006de50 con 0x7f41b00c93d0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f41b006ed40 con 0x7f41b006d9d0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.566+0000 7f41b5de4640 10 monclient(hunting): authenticate will time out at 1133.146561s 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41affff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f41b00c93d0 0x7f41b0158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:32952/0 (socket says 10.20.193.51:32952) 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41affff640 1 -- 10.20.193.51:0/2489353005 learned_addr learned my addr 10.20.193.51:0/2489353005 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 1 --2- 10.20.193.51:0/2489353005 >> v2:10.20.193.68:3300/0 conn(0x7f41b006d9d0 0x7f41b00c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): get_auth_request con 0x7f41b006d9d0 auth_method 0 2026-02-22T21:37:37.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.567+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 1 -- 10.20.193.51:0/2489353005 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255048383 0 0) 0x7f41b006de50 con 0x7f41b00c93d0 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 10 monclient(hunting): my global_id is 4388 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41ae7fc640 1 -- 10.20.193.51:0/2489353005 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4194003620 con 0x7f41b00c93d0 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 10 monclient(hunting): handle_auth_done global_id 4402 payload 274 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 1 -- 10.20.193.51:0/2489353005 >> v1:10.20.193.125:6789/0 conn(0x7f41b00c93d0 legacy=0x7f41b0158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 1 -- 10.20.193.51:0/2489353005 >> v1:10.20.193.51:6789/0 conn(0x7f41b0152ab0 legacy=0x7f41b00c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:37.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 1 monclient: found mon.noname-b 2026-02-22T21:37:37.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:37.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.568+0000 7f41aeffd640 1 -- 10.20.193.51:0/2489353005 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b006e130 con 0x7f41b006d9d0 2026-02-22T21:37:37.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.569+0000 7f41aeffd640 1 --2- 10.20.193.51:0/2489353005 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c8e90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41a00098d0 tx=0x7f41a00311b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab885ad6dbe9044a server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 1 -- 10.20.193.51:0/2489353005 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a003e070 con 0x7f41b006d9d0 2026-02-22T21:37:37.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: dump: 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 1 -- 10.20.193.51:0/2489353005 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a0039440 con 0x7f41b006d9d0 2026-02-22T21:37:37.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 1 -- 10.20.193.51:0/2489353005 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0039810 con 0x7f41b006d9d0 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41ae7fc640 10 monclient: dump: 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 5 monclient: authenticate success, global_id 4402 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 4 set_mon_vals no callback set 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 10 monclient: shutdown 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 -- 10.20.193.51:0/2489353005 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 msgr2=0x7f41b00c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2489353005 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c8e90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f41a00098d0 tx=0x7f41a00311b0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 -- 10.20.193.51:0/2489353005 shutdown_connections 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2489353005 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c8e90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 -- 10.20.193.51:0/2489353005 >> 10.20.193.51:0/2489353005 conn(0x7f41b00cd7b0 msgr2=0x7f41b00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 -- 10.20.193.51:0/2489353005 shutdown_connections 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.571+0000 7f41b5de4640 1 -- 10.20.193.51:0/2489353005 wait complete. 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 10 monclient: build_initial_monmap 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 10 monclient: monmap: 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:37.579 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 librados: starting msgr at 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 librados: starting objecter 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 Processor -- start 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 -- start start 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 librados: setting wanted keys 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 librados: calling monclient init 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 10 monclient: init 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.572+0000 7f41b5de4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-b con 0x7f41b006d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f41b00c93d0 0x7f41b00732f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-a con 0x7f41b00c93d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f41b0152ab0 0x7f41b0073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): picked mon.noname-c con 0x7f41b0152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f41b006e6f0 con 0x7f41b00c93d0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f41b00c9880 con 0x7f41b006d9d0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f41b006e9d0 con 0x7f41b0152ab0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41b5de4640 10 monclient(hunting): authenticate will time out at 1133.153035s 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f41b00c93d0 0x7f41b00732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41affff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f41b0152ab0 0x7f41b0073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35200/0 (socket says 10.20.193.51:35200) 2026-02-22T21:37:37.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 1 -- 10.20.193.51:0/2289473836 learned_addr learned my addr 10.20.193.51:0/2289473836 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): get_auth_request con 0x7f41b006d9d0 auth_method 0 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): get_auth_request con 0x7f41b00c93d0 auth_method 0 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41affff640 10 monclient(hunting): get_auth_request con 0x7f41b0152ab0 auth_method 0 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41affff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.573+0000 7f41aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.574+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.574+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.574+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 -- 10.20.193.51:0/2289473836 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f41b0152ab0 msgr2=0x7f41b0073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f41b0152ab0 0x7f41b0073830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 -- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f41b00c93d0 msgr2=0x7f41b00732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f41b00c93d0 0x7f41b00732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.575+0000 7f41af7fe640 1 -- 10.20.193.51:0/2289473836 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b00c3f80 con 0x7f41b006d9d0 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.576+0000 7f41af7fe640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c700 tx=0x7f419c00cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.576+0000 7f418bfff640 10 client.?.objecter ms_handle_connect 0x7f41b006d9d0 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.576+0000 7f418bfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c0052d0 con 0x7f41b006d9d0 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: dump: 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.585 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 5 monclient: authenticate success, global_id 4405 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 10 monclient: _renew_subs 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b0072530 con 0x7f41b006d9d0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 10 monclient: _renew_subs 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b0072af0 con 0x7f41b006d9d0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41b5de4640 1 librados: init done 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c005470 con 0x7f41b006d9d0 2026-02-22T21:37:37.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c00a840 con 0x7f41b006d9d0 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f41adffb640 4 set_mon_vals no callback set 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.577+0000 7f418bfff640 10 monclient: dump: 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.578+0000 7f41b5de4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.578+0000 7f41b5de4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.578+0000 7f41b5de4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.578+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b0072e20 con 0x7f41b006d9d0 2026-02-22T21:37:37.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f419c00a9e0 con 0x7f41b006d9d0 2026-02-22T21:37:37.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f418bfff640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 0x7f418007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f41aeffd640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 0x7f418007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f41aeffd640 10 monclient: get_auth_request con 0x7f4180078e00 auth_method 0 2026-02-22T21:37:37.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f418bfff640 10 client.4405.objecter ms_handle_connect 0x7f4180078e00 2026-02-22T21:37:37.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.687+0000 7f41aeffd640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 0x7f418007b2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41a003b000 tx=0x7f41a0035000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.689+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f419c00ad00 con 0x7f41b006d9d0 2026-02-22T21:37:37.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.689+0000 7f418bfff640 10 client.4405.objecter ms_dispatch2 0x7f41b0000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:37.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.689+0000 7f418bfff640 3 client.4405.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:37.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.689+0000 7f418bfff640 3 client.4405.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:37.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.689+0000 7f418bfff640 20 client.4405.objecter dump_active .. 0 homeless 2026-02-22T21:37:37.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.696+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f419c062030 con 0x7f41b006d9d0 2026-02-22T21:37:37.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.696+0000 7f418bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.696+0000 7f418bfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:37.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.779+0000 7f41b5de4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:37.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.779+0000 7f41b5de4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:37.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.779+0000 7f41b5de4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.779+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41b0152e90 con 0x7f41b006d9d0 2026-02-22T21:37:37.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f418bfff640 1 -- 10.20.193.51:0/2289473836 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+24729 (secure 0 0 0) 0x7f419c066330 con 0x7f41b006d9d0 2026-02-22T21:37:37.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f418bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:37.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f418bfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:37.805 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:37.805 INFO:teuthology.orchestra.run.trial051.stdout:{"epoch":30,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:37:36.156780+0000","last_up_change":"2026-02-22T21:37:29.474866+0000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T21:37:27.901086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T21:37:30.749692+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6802","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6803","nonce":1198482343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6804","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6805","nonce":1198482343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6808","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6809","nonce":1198482343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6806","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6807","nonce":1198482343}]},"public_addr":"10.20.193.51:6803/1198482343","cluster_addr":"10.20.193.51:6805/1198482343","heartbeat_back_addr":"10.20.193.51:6809/1198482343","heartbeat_front_addr":"10.20.193.51:6807/1198482343","state":["exists","up"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6825","nonce":2540990948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6827","nonce":2540990948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6831","nonce":2540990948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6828","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6829","nonce":2540990948}]},"public_addr":"10.20.193.68:6825/2540990948","cluster_addr":"10.20.193.68:6827/2540990948","heartbeat_back_addr":"10.20.193.68:6831/2540990948","heartbeat_front_addr":"10.20.193.68:6829/2540990948","state":["exists","up"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6808","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6809","nonce":1963822926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6810","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6811","nonce":1963822926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6814","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6815","nonce":1963822926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6812","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6813","nonce":1963822926}]},"public_addr":"10.20.193.125:6809/1963822926","cluster_addr":"10.20.193.125:6811/1963822926","heartbeat_back_addr":"10.20.193.125:6815/1963822926","heartbeat_front_addr":"10.20.193.125:6813/1963822926","state":["exists","up"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6824","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6825","nonce":3567184175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6826","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6827","nonce":3567184175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6830","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6831","nonce":3567184175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6828","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6829","nonce":3567184175}]},"public_addr":"10.20.193.140:6825/3567184175","cluster_addr":"10.20.193.140:6827/3567184175","heartbeat_back_addr":"10.20.193.140:6831/3567184175","heartbeat_front_addr":"10.20.193.140:6829/3567184175","state":["exists","up"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6810","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6811","nonce":1021106712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6812","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6813","nonce":1021106712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6816","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6817","nonce":1021106712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6814","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6815","nonce":1021106712}]},"public_addr":"10.20.193.51:6811/1021106712","cluster_addr":"10.20.193.51:6813/1021106712","heartbeat_back_addr":"10.20.193.51:6817/1021106712","heartbeat_front_addr":"10.20.193.51:6815/1021106712","state":["exists","up"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6817","nonce":3736303751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6819","nonce":3736303751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6823","nonce":3736303751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6821","nonce":3736303751}]},"public_addr":"10.20.193.68:6817/3736303751","cluster_addr":"10.20.193.68:6819/3736303751","heartbeat_back_addr":"10.20.193.68:6823/3736303751","heartbeat_front_addr":"10.20.193.68:6821/3736303751","state":["exists","up"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6824","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6825","nonce":1184344866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6826","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6827","nonce":1184344866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6830","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6831","nonce":1184344866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6828","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6829","nonce":1184344866}]},"public_addr":"10.20.193.125:6825/1184344866","cluster_addr":"10.20.193.125:6827/1184344866","heartbeat_back_addr":"10.20.193.125:6831/1184344866","heartbeat_front_addr":"10.20.193.125:6829/1184344866","state":["exists","up"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6808","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6809","nonce":1140578699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6810","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6811","nonce":1140578699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6814","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6815","nonce":1140578699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6812","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6813","nonce":1140578699}]},"public_addr":"10.20.193.140:6809/1140578699","cluster_addr":"10.20.193.140:6811/1140578699","heartbeat_back_addr":"10.20.193.140:6815/1140578699","heartbeat_front_addr":"10.20.193.140:6813/1140578699","state":["exists","up"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6826","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6827","nonce":142198174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6828","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6829","nonce":142198174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6832","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6833","nonce":142198174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6830","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6831","nonce":142198174}]},"public_addr":"10.20.193.51:6827/142198174","cluster_addr":"10.20.193.51:6829/142198174","heartbeat_back_addr":"10.20.193.51:6833/142198174","heartbeat_front_addr":"10.20.193.51:6831/142198174","state":["exists","up"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6809","nonce":1841964582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6811","nonce":1841964582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6815","nonce":1841964582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6813","nonce":1841964582}]},"public_addr":"10.20.193.68:6809/1841964582","cluster_addr":"10.20.193.68:6811/1841964582","heartbeat_back_addr":"10.20.193.68:6815/1841964582","heartbeat_front_addr":"10.20.193.68:6813/1841964582","state":["exists","up"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6800","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6801","nonce":1951583296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6802","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6803","nonce":1951583296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6806","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6807","nonce":1951583296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6804","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6805","nonce":1951583296}]},"public_addr":"10.20.193.125:6801/1951583296","cluster_addr":"10.20.193.125:6803/1951583296","heartbeat_back_addr":"10.20.193.125:6807/1951583296","heartbeat_front_addr":"10.20.193.125:6805/1951583296","state":["exists","up"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6816","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6817","nonce":2298497951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6818","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6819","nonce":2298497951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6822","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6823","nonce":2298497951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6820","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6821","nonce":2298497951}]},"public_addr":"10.20.193.140:6817/2298497951","cluster_addr":"10.20.193.140:6819/2298497951","heartbeat_back_addr":"10.20.193.140:6823/2298497951","heartbeat_front_addr":"10.20.193.140:6821/2298497951","state":["exists","up"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6818","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6819","nonce":1676524147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6820","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6821","nonce":1676524147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6824","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6825","nonce":1676524147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6822","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6823","nonce":1676524147}]},"public_addr":"10.20.193.51:6819/1676524147","cluster_addr":"10.20.193.51:6821/1676524147","heartbeat_back_addr":"10.20.193.51:6825/1676524147","heartbeat_front_addr":"10.20.193.51:6823/1676524147","state":["exists","up"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6801","nonce":762840177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6803","nonce":762840177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6807","nonce":762840177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6805","nonce":762840177}]},"public_addr":"10.20.193.68:6801/762840177","cluster_addr":"10.20.193.68:6803/762840177","heartbeat_back_addr":"10.20.193.68:6807/762840177","heartbeat_front_addr":"10.20.193.68:6805/762840177","state":["exists","up"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6816","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6817","nonce":498199928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6818","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6819","nonce":498199928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6822","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6823","nonce":498199928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6820","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6821","nonce":498199928}]},"public_addr":"10.20.193.125:6817/498199928","cluster_addr":"10.20.193.125:6819/498199928","heartbeat_back_addr":"10.20.193.125:6823/498199928","heartbeat_front_addr":"10.20.193.125:6821/498199928","state":["exists","up"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6800","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6801","nonce":2657947218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6802","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6803","nonce":2657947218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6806","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6807","nonce":2657947218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6804","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6805","nonce":2657947218}]},"public_addr":"10.20.193.140:6801/2657947218","cluster_addr":"10.20.193.140:6803/2657947218","heartbeat_back_addr":"10.20.193.140:6807/2657947218","heartbeat_front_addr":"10.20.193.140:6805/2657947218","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.521455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.773204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.592098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.942138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.663559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.647770+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.811906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.107957+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.279951+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.927867+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.306670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.794057+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.055278+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.699761+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.177807+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:24.554387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:37.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 10 librados: watch_flush enter 2026-02-22T21:37:37.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 10 librados: watch_flush exit 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 20 client.4405.objecter shutdown clearing up homeless session... 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 10 client.4405.objecter successfully canceled tick 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 msgr2=0x7f418007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 0x7f418007b2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41a003b000 tx=0x7f41a0035000 comp rx=0 tx=0).stop 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 10 monclient: shutdown 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 msgr2=0x7f41b00c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.802+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c700 tx=0x7f419c00cbd0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 shutdown_connections 2026-02-22T21:37:37.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4180078e00 0x7f418007b2c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f41b0152ab0 0x7f41b0073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f41b00c93d0 0x7f41b00732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 --2- 10.20.193.51:0/2289473836 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f41b006d9d0 0x7f41b00c1340 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 >> 10.20.193.51:0/2289473836 conn(0x7f41b00cd7b0 msgr2=0x7f41b00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 shutdown_connections 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 -- 10.20.193.51:0/2289473836 wait complete. 2026-02-22T21:37:37.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.803+0000 7f41b5de4640 1 librados: shutdown 2026-02-22T21:37:37.813 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-22T21:37:37.813 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T21:37:37.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.878+0000 7f1e36274640 10 monclient: get_monmap_and_config 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.878+0000 7f1e36274640 10 monclient: build_initial_monmap 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.878+0000 7f1e36274640 10 monclient: monmap: 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:37.879619+0000 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:37.879619+0000 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.881 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 Processor -- start 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 -- start start 2026-02-22T21:37:37.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient: init 2026-02-22T21:37:37.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient: _add_conns ranks=[0,1,3,2,5,4] 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e30150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f1e3015c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1e300703b0 0x7f1e300c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-a con 0x7f1e300703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e3015a920 con 0x7f1e30150220 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e3015af60 con 0x7f1e3015c710 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f1e3015b240 con 0x7f1e300703b0 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:37.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.879+0000 7f1e36274640 10 monclient(hunting): authenticate will time out at 1133.459924s 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2effd640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f1e3015c710 0x7f1e3006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54312/0 (socket says 10.20.193.51:54312) 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2effd640 1 -- 10.20.193.51:0/767206015 learned_addr learned my addr 10.20.193.51:0/767206015 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2ffff640 1 --2- 10.20.193.51:0/767206015 >> v2:10.20.193.51:3300/0 conn(0x7f1e300703b0 0x7f1e300c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request con 0x7f1e300703b0 auth_method 0 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.880+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 1 -- 10.20.193.51:0/767206015 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225424730 0 0) 0x7f1e3015af60 con 0x7f1e3015c710 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 10 monclient(hunting): my global_id is 4408 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.881+0000 7f1e2e7fc640 1 -- 10.20.193.51:0/767206015 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e14003620 con 0x7f1e3015c710 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 1 -- 10.20.193.51:0/767206015 >> v1:10.20.193.68:6789/0 conn(0x7f1e3015c710 legacy=0x7f1e3006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 1 -- 10.20.193.51:0/767206015 >> v1:10.20.193.51:6789/0 conn(0x7f1e30150220 legacy=0x7f1e3015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 1 monclient: found mon.noname-a 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:37.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:37.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.882+0000 7f1e2ffff640 1 -- 10.20.193.51:0/767206015 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3015b520 con 0x7f1e300703b0 2026-02-22T21:37:37.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.885+0000 7f1e2ffff640 1 --2- 10.20.193.51:0/767206015 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e300703b0 0x7f1e300c73a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009dc0 tx=0x7f1e2402f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f8d7d1865f4048 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 1 -- 10.20.193.51:0/767206015 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2403c070 con 0x7f1e300703b0 2026-02-22T21:37:37.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: dump: 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.893 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 1 -- 10.20.193.51:0/767206015 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e24037440 con 0x7f1e300703b0 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 1 -- 10.20.193.51:0/767206015 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e24037810 con 0x7f1e300703b0 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 5 monclient: authenticate success, global_id 4398 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 4 set_mon_vals no callback set 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:37.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e2e7fc640 10 monclient: dump: 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 10 monclient: shutdown 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 -- 10.20.193.51:0/767206015 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e300703b0 msgr2=0x7f1e300c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 --2- 10.20.193.51:0/767206015 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e300703b0 0x7f1e300c73a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009dc0 tx=0x7f1e2402f2f0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 -- 10.20.193.51:0/767206015 shutdown_connections 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 --2- 10.20.193.51:0/767206015 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e300703b0 0x7f1e300c73a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 -- 10.20.193.51:0/767206015 >> 10.20.193.51:0/767206015 conn(0x7f1e300cd7b0 msgr2=0x7f1e300cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 -- 10.20.193.51:0/767206015 shutdown_connections 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.887+0000 7f1e36274640 1 -- 10.20.193.51:0/767206015 wait complete. 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 10 monclient: build_initial_monmap 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 10 monclient: monmap: 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:37.896 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 librados: starting msgr at 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 librados: starting objecter 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 Processor -- start 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 -- start start 2026-02-22T21:37:37.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 librados: setting wanted keys 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 1 librados: calling monclient init 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.888+0000 7f1e36274640 10 monclient: init 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1e300703b0 0x7f1e301f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-c con 0x7f1e300703b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 0x7f1e300bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-b con 0x7f1e30150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 0x7f1e301e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): picked mon.noname-a con 0x7f1e3015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30150600 con 0x7f1e3015c710 2026-02-22T21:37:37.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3015bae0 con 0x7f1e30150220 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30150780 con 0x7f1e300703b0 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e36274640 10 monclient(hunting): authenticate will time out at 1133.469353s 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 0x7f1e301e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2effd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 0x7f1e300bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2f7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1e300703b0 0x7f1e301f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 0x7f1e301e4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60646/0 (socket says 10.20.193.51:60646) 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 1 -- 10.20.193.51:0/2954971615 learned_addr learned my addr 10.20.193.51:0/2954971615 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request con 0x7f1e3015c710 auth_method 0 2026-02-22T21:37:37.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.889+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): get_auth_request con 0x7f1e30150220 auth_method 0 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): get_auth_request con 0x7f1e300703b0 auth_method 0 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:37.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:37.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.890+0000 7f1e2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:37.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 -- 10.20.193.51:0/2954971615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1e300703b0 msgr2=0x7f1e301f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1e300703b0 0x7f1e301f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 -- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 msgr2=0x7f1e301e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 0x7f1e301e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 monclient: found mon.noname-b 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.891+0000 7f1e2effd640 1 -- 10.20.193.51:0/2954971615 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300c0170 con 0x7f1e30150220 2026-02-22T21:37:37.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.892+0000 7f1e2effd640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 0x7f1e300bfc30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c4d0 tx=0x7f1e2000c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.892+0000 7f1e0bfff640 10 client.?.objecter ms_handle_connect 0x7f1e30150220 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.892+0000 7f1e0bfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20017070 con 0x7f1e30150220 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: dump: 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.905 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e20005de0 con 0x7f1e30150220 2026-02-22T21:37:37.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 5 monclient: authenticate success, global_id 4411 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 10 monclient: _renew_subs 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e300be8b0 con 0x7f1e30150220 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20007290 con 0x7f1e30150220 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e2dffb640 4 set_mon_vals no callback set 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e0bfff640 10 monclient: dump: 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:37.907 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 10 monclient: _renew_subs 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e300bee70 con 0x7f1e30150220 2026-02-22T21:37:37.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.894+0000 7f1e36274640 1 librados: init done 2026-02-22T21:37:37.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.895+0000 7f1e36274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.895+0000 7f1e36274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:37.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.895+0000 7f1e36274640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:37.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.895+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e30074da0 con 0x7f1e30150220 2026-02-22T21:37:37.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.906+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1e20007430 con 0x7f1e30150220 2026-02-22T21:37:37.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.906+0000 7f1e0bfff640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 0x7f1e0007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:37.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.907+0000 7f1e2f7fe640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 0x7f1e0007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.907+0000 7f1e2f7fe640 10 monclient: get_auth_request con 0x7f1e00078e00 auth_method 0 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.907+0000 7f1e2f7fe640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 0x7f1e0007b2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c002410 tx=0x7f1e1c007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.907+0000 7f1e0bfff640 10 client.4411.objecter ms_handle_connect 0x7f1e00078e00 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.909+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f1e2005d3e0 con 0x7f1e30150220 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.909+0000 7f1e0bfff640 10 client.4411.objecter ms_dispatch2 0x7f1e30000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.909+0000 7f1e0bfff640 3 client.4411.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.909+0000 7f1e0bfff640 3 client.4411.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:37.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:37.909+0000 7f1e0bfff640 20 client.4411.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.011+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f1e20021020 con 0x7f1e30150220 2026-02-22T21:37:38.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.011+0000 7f1e0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.011+0000 7f1e0bfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.094+0000 7f1e36274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:38.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.094+0000 7f1e36274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:38.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.094+0000 7f1e36274640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.094+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e3014e280 con 0x7f1e30150220 2026-02-22T21:37:38.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.113+0000 7f1e0bfff640 1 -- 10.20.193.51:0/2954971615 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+24729 (secure 0 0 0) 0x7f1e200613a0 con 0x7f1e30150220 2026-02-22T21:37:38.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.113+0000 7f1e0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:37:38.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.113+0000 7f1e0bfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:38.114 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:38.114 INFO:teuthology.orchestra.run.trial051.stdout:{"epoch":30,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:37:36.156780+0000","last_up_change":"2026-02-22T21:37:29.474866+0000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T21:37:27.901086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T21:37:30.749692+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6802","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6803","nonce":1198482343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6804","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6805","nonce":1198482343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6808","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6809","nonce":1198482343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6806","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6807","nonce":1198482343}]},"public_addr":"10.20.193.51:6803/1198482343","cluster_addr":"10.20.193.51:6805/1198482343","heartbeat_back_addr":"10.20.193.51:6809/1198482343","heartbeat_front_addr":"10.20.193.51:6807/1198482343","state":["exists","up"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6825","nonce":2540990948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6827","nonce":2540990948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6831","nonce":2540990948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6828","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6829","nonce":2540990948}]},"public_addr":"10.20.193.68:6825/2540990948","cluster_addr":"10.20.193.68:6827/2540990948","heartbeat_back_addr":"10.20.193.68:6831/2540990948","heartbeat_front_addr":"10.20.193.68:6829/2540990948","state":["exists","up"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6808","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6809","nonce":1963822926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6810","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6811","nonce":1963822926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6814","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6815","nonce":1963822926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6812","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6813","nonce":1963822926}]},"public_addr":"10.20.193.125:6809/1963822926","cluster_addr":"10.20.193.125:6811/1963822926","heartbeat_back_addr":"10.20.193.125:6815/1963822926","heartbeat_front_addr":"10.20.193.125:6813/1963822926","state":["exists","up"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6824","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6825","nonce":3567184175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6826","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6827","nonce":3567184175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6830","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6831","nonce":3567184175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6828","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6829","nonce":3567184175}]},"public_addr":"10.20.193.140:6825/3567184175","cluster_addr":"10.20.193.140:6827/3567184175","heartbeat_back_addr":"10.20.193.140:6831/3567184175","heartbeat_front_addr":"10.20.193.140:6829/3567184175","state":["exists","up"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6810","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6811","nonce":1021106712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6812","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6813","nonce":1021106712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6816","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6817","nonce":1021106712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6814","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6815","nonce":1021106712}]},"public_addr":"10.20.193.51:6811/1021106712","cluster_addr":"10.20.193.51:6813/1021106712","heartbeat_back_addr":"10.20.193.51:6817/1021106712","heartbeat_front_addr":"10.20.193.51:6815/1021106712","state":["exists","up"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6817","nonce":3736303751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6819","nonce":3736303751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6823","nonce":3736303751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6821","nonce":3736303751}]},"public_addr":"10.20.193.68:6817/3736303751","cluster_addr":"10.20.193.68:6819/3736303751","heartbeat_back_addr":"10.20.193.68:6823/3736303751","heartbeat_front_addr":"10.20.193.68:6821/3736303751","state":["exists","up"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6824","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6825","nonce":1184344866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6826","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6827","nonce":1184344866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6830","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6831","nonce":1184344866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6828","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6829","nonce":1184344866}]},"public_addr":"10.20.193.125:6825/1184344866","cluster_addr":"10.20.193.125:6827/1184344866","heartbeat_back_addr":"10.20.193.125:6831/1184344866","heartbeat_front_addr":"10.20.193.125:6829/1184344866","state":["exists","up"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6808","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6809","nonce":1140578699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6810","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6811","nonce":1140578699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6814","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6815","nonce":1140578699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6812","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6813","nonce":1140578699}]},"public_addr":"10.20.193.140:6809/1140578699","cluster_addr":"10.20.193.140:6811/1140578699","heartbeat_back_addr":"10.20.193.140:6815/1140578699","heartbeat_front_addr":"10.20.193.140:6813/1140578699","state":["exists","up"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6826","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6827","nonce":142198174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6828","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6829","nonce":142198174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6832","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6833","nonce":142198174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6830","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6831","nonce":142198174}]},"public_addr":"10.20.193.51:6827/142198174","cluster_addr":"10.20.193.51:6829/142198174","heartbeat_back_addr":"10.20.193.51:6833/142198174","heartbeat_front_addr":"10.20.193.51:6831/142198174","state":["exists","up"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6809","nonce":1841964582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6811","nonce":1841964582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6815","nonce":1841964582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6813","nonce":1841964582}]},"public_addr":"10.20.193.68:6809/1841964582","cluster_addr":"10.20.193.68:6811/1841964582","heartbeat_back_addr":"10.20.193.68:6815/1841964582","heartbeat_front_addr":"10.20.193.68:6813/1841964582","state":["exists","up"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6800","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6801","nonce":1951583296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6802","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6803","nonce":1951583296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6806","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6807","nonce":1951583296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6804","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6805","nonce":1951583296}]},"public_addr":"10.20.193.125:6801/1951583296","cluster_addr":"10.20.193.125:6803/1951583296","heartbeat_back_addr":"10.20.193.125:6807/1951583296","heartbeat_front_addr":"10.20.193.125:6805/1951583296","state":["exists","up"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6816","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6817","nonce":2298497951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6818","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6819","nonce":2298497951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6822","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6823","nonce":2298497951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6820","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6821","nonce":2298497951}]},"public_addr":"10.20.193.140:6817/2298497951","cluster_addr":"10.20.193.140:6819/2298497951","heartbeat_back_addr":"10.20.193.140:6823/2298497951","heartbeat_front_addr":"10.20.193.140:6821/2298497951","state":["exists","up"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6818","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6819","nonce":1676524147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6820","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6821","nonce":1676524147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6824","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6825","nonce":1676524147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6822","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6823","nonce":1676524147}]},"public_addr":"10.20.193.51:6819/1676524147","cluster_addr":"10.20.193.51:6821/1676524147","heartbeat_back_addr":"10.20.193.51:6825/1676524147","heartbeat_front_addr":"10.20.193.51:6823/1676524147","state":["exists","up"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6801","nonce":762840177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6803","nonce":762840177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6807","nonce":762840177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6805","nonce":762840177}]},"public_addr":"10.20.193.68:6801/762840177","cluster_addr":"10.20.193.68:6803/762840177","heartbeat_back_addr":"10.20.193.68:6807/762840177","heartbeat_front_addr":"10.20.193.68:6805/762840177","state":["exists","up"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6816","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6817","nonce":498199928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6818","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6819","nonce":498199928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6822","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6823","nonce":498199928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6820","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6821","nonce":498199928}]},"public_addr":"10.20.193.125:6817/498199928","cluster_addr":"10.20.193.125:6819/498199928","heartbeat_back_addr":"10.20.193.125:6823/498199928","heartbeat_front_addr":"10.20.193.125:6821/498199928","state":["exists","up"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6800","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6801","nonce":2657947218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6802","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6803","nonce":2657947218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6806","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6807","nonce":2657947218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6804","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6805","nonce":2657947218}]},"public_addr":"10.20.193.140:6801/2657947218","cluster_addr":"10.20.193.140:6803/2657947218","heartbeat_back_addr":"10.20.193.140:6807/2657947218","heartbeat_front_addr":"10.20.193.140:6805/2657947218","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.521455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.773204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.592098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.942138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.663559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.647770+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.811906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.107957+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.279951+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.927867+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.306670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.794057+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.055278+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.699761+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.177807+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:24.554387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 10 librados: watch_flush enter 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 10 librados: watch_flush exit 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 20 client.4411.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 10 client.4411.objecter successfully canceled tick 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 msgr2=0x7f1e0007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 0x7f1e0007b2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c002410 tx=0x7f1e1c007ef0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 10 monclient: shutdown 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 msgr2=0x7f1e300bfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 0x7f1e300bfc30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c4d0 tx=0x7f1e2000c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 shutdown_connections 2026-02-22T21:37:38.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1e00078e00 0x7f1e0007b2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1e3015c710 0x7f1e301e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1e30150220 0x7f1e300bfc30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 --2- 10.20.193.51:0/2954971615 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1e300703b0 0x7f1e301f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 >> 10.20.193.51:0/2954971615 conn(0x7f1e300cd7b0 msgr2=0x7f1e300cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 shutdown_connections 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 -- 10.20.193.51:0/2954971615 wait complete. 2026-02-22T21:37:38.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.114+0000 7f1e36274640 1 librados: shutdown 2026-02-22T21:37:38.124 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T21:37:38.124 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T21:37:38.124 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T21:37:38.124 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T21:37:38.124 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2026-02-22T21:37:38.125 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2026-02-22T21:37:38.126 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2026-02-22T21:37:38.126 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2026-02-22T21:37:38.126 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2026-02-22T21:37:38.126 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2026-02-22T21:37:38.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 10 monclient: monmap: 2026-02-22T21:37:38.206 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.204956+0000 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.204956+0000 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.207 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 1 Processor -- start 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 1 -- start start 2026-02-22T21:37:38.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.204+0000 7ff605355640 10 monclient: init 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient: _add_conns ranks=[4,2,1,5,3,0] 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ff600150220 0x7ff6000c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): picked mon.noname-b con 0x7ff600150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff600150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7ff60015c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff60015a920 con 0x7ff60015c710 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff60015af60 con 0x7ff600150600 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7ff60015b240 con 0x7ff600150220 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7ff605355640 10 monclient(hunting): authenticate will time out at 1133.785508s 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7f2e897ce640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.205+0000 7f2e897ce640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7f2e897ce640 10 monclient: monmap: 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.206700+0000 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.206700+0000 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.212 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5fe7fc640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7ff600150600 0x7ff60015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:32964/0 (socket says 10.20.193.51:32964) 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 10 monclient: monmap: 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.206734+0000 2026-02-22T21:37:38.214 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.206734+0000 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.215 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5fe7fc640 1 -- 10.20.193.51:0/3534233387 learned_addr learned my addr 10.20.193.51:0/3534233387 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5ff7fe640 1 --1- 10.20.193.51:0/3534233387 >> v1:10.20.193.68:6789/0 conn(0x7ff60015c710 0x7ff60006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54316/0 (socket says 10.20.193.51:54316) 2026-02-22T21:37:38.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 1 --2- 10.20.193.51:0/3534233387 >> v2:10.20.193.68:3300/0 conn(0x7ff600150220 0x7ff6000c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): get_auth_request con 0x7ff600150220 auth_method 0 2026-02-22T21:37:38.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7f2e897ce640 1 Processor -- start 2026-02-22T21:37:38.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 1 Processor -- start 2026-02-22T21:37:38.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7f2e897ce640 1 -- start start 2026-02-22T21:37:38.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7f2e897ce640 10 monclient: init 2026-02-22T21:37:38.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 1 -- start start 2026-02-22T21:37:38.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7fc9a1d85640 10 monclient: init 2026-02-22T21:37:38.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.206+0000 7ff5feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient: _add_conns ranks=[0,1,2,5,4,3] 2026-02-22T21:37:38.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient: _add_conns ranks=[0,5,4,1,3,2] 2026-02-22T21:37:38.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e84155e60 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc99c1526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fc99c152a80 0x7fc99c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c152a80 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc99c0cdb20 0x7fc99c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-b con 0x7fc99c0cdb20 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc99c073ae0 con 0x7fc99c1526a0 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc99c0cca00 con 0x7fc99c0cdb20 2026-02-22T21:37:38.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fc99c074c50 con 0x7fc99c152a80 2026-02-22T21:37:38.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fc9a1d85640 10 monclient(hunting): authenticate will time out at 1133.787299s 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f2e8415c790 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e840c9470 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e840cbc80 con 0x7f2e84155e60 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e840cc450 con 0x7f2e8415c790 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e840cc730 con 0x7f2e840c9470 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f2e897ce640 10 monclient(hunting): authenticate will time out at 1133.787440s 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 10 monclient: monmap: 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.208075+0000 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.208075+0000 2026-02-22T21:37:38.225 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.226 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 10 monclient(hunting): handle_auth_done global_id 4414 payload 274 2026-02-22T21:37:38.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 1 -- 10.20.193.51:0/3534233387 >> v1:10.20.193.125:6789/0 conn(0x7ff600150600 legacy=0x7ff60015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 1 -- 10.20.193.51:0/3534233387 >> v1:10.20.193.68:6789/0 conn(0x7ff60015c710 legacy=0x7ff60006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7ff5feffd640 1 -- 10.20.193.51:0/3534233387 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60015b520 con 0x7ff600150220 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f73e1f5c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f73e1f5c640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7f73e1f5c640 10 monclient: monmap: 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.208443+0000 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.208443+0000 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.229 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.230 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 1 Processor -- start 2026-02-22T21:37:38.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 1 -- start start 2026-02-22T21:37:38.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.207+0000 7fbc0b3d5640 10 monclient: init 2026-02-22T21:37:38.231 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99b7fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fc99c1526a0 0x7fc99c0733d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46882/0 (socket says 10.20.193.51:46882) 2026-02-22T21:37:38.231 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99b7fe640 1 -- 10.20.193.51:0/4056165522 learned_addr learned my addr 10.20.193.51:0/4056165522 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.231 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f2e82ffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f2e84155e60 0x7f2e840cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46876/0 (socket says 10.20.193.51:46876) 2026-02-22T21:37:38.231 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f2e82ffd640 1 -- 10.20.193.51:0/63750042 learned_addr learned my addr 10.20.193.51:0/63750042 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 10 monclient: monmap: 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.208811+0000 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.208811+0000 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.232 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99affd640 1 --2- 10.20.193.51:0/4056165522 >> v2:10.20.193.125:3300/0 conn(0x7fc99c152a80 0x7fc99c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 1 Processor -- start 2026-02-22T21:37:38.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 1 -- start start 2026-02-22T21:37:38.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 10 monclient: init 2026-02-22T21:37:38.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99affd640 10 monclient(hunting): get_auth_request con 0x7fc99c152a80 auth_method 0 2026-02-22T21:37:38.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fc99affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient: _add_conns ranks=[0,4,3,5,1,2] 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc041526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbc04152a80 0x7fbc040cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-b con 0x7fbc04152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbc040cdb20 0x7fbc0415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-a con 0x7fbc040cdb20 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc04073ae0 con 0x7fbc041526a0 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fbc040cca00 con 0x7fbc040cdb20 2026-02-22T21:37:38.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fbc04074c50 con 0x7fbc04152a80 2026-02-22T21:37:38.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7fbc0b3d5640 10 monclient(hunting): authenticate will time out at 1133.788648s 2026-02-22T21:37:38.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 1 Processor -- start 2026-02-22T21:37:38.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 10 monclient: _add_conns ranks=[4,2,5,3,0,1] 2026-02-22T21:37:38.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f73dc152cc0 0x7f73dc0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-b con 0x7f73dc152cc0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73dc0c58b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f73dc15aec0 0x7f73dc158af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc15aec0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73dc159ff0 con 0x7f73dc0c58b0 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc074f90 con 0x7f73dc152cc0 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 1 -- start start 2026-02-22T21:37:38.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.208+0000 7f7c45643640 10 monclient: init 2026-02-22T21:37:38.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc0c5cf0 con 0x7f73dc15aec0 2026-02-22T21:37:38.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f73e1f5c640 10 monclient(hunting): authenticate will time out at 1133.789033s 2026-02-22T21:37:38.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 10 monclient: monmap: 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.209841+0000 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.209841+0000 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.241 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.242 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc99affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc99affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc99affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient: _add_conns ranks=[1,3,0,4,2,5] 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f7c40155e60 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7c4015c790 0x7f7c400c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4015c790 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c400c92a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c400c9710 con 0x7f7c400c92a0 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c400cbc80 con 0x7f7c40155e60 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f7c400cc200 con 0x7f7c4015c790 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7f7c45643640 10 monclient(hunting): authenticate will time out at 1133.789492s 2026-02-22T21:37:38.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7ff5feffd640 1 --2- 10.20.193.51:0/3534233387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff600150220 0x7ff6000c73a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40098a0 tx=0x7ff5f402ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1216b6975a00a280 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 1 Processor -- start 2026-02-22T21:37:38.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 1 -- start start 2026-02-22T21:37:38.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.209+0000 7fc17d944640 10 monclient: init 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 1 -- 10.20.193.51:0/4056165522 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107043555 0 0) 0x7fc988002920 con 0x7fc99c1526a0 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 10 monclient(hunting): my global_id is 4404 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc99a7fc640 1 -- 10.20.193.51:0/4056165522 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc980003620 con 0x7fc99c1526a0 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7f101ef11640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7f101ef11640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7f101ef11640 10 monclient: monmap: 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.245 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.211270+0000 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.211270+0000 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.246 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient: _add_conns ranks=[5,4,2,1,3,0] 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fc178153970 0x7fc178153d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-c con 0x7fc178153970 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc17815a420 0x7fc17806f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-b con 0x7fc17815a420 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc17806d710 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc178071040 con 0x7fc17806d710 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc17806f880 con 0x7fc17815a420 2026-02-22T21:37:38.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fc1780c4d40 con 0x7fc178153970 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.210+0000 7fc17d944640 10 monclient(hunting): authenticate will time out at 1133.790755s 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 Processor -- start 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 -- start start 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient: init 2026-02-22T21:37:38.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbc04152a80 0x7fbc040cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient: _add_conns ranks=[2,3,4,5,1,0] 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 1 -- 10.20.193.51:0/4056165522 >> v2:10.20.193.68:3300/0 conn(0x7fc99c0cdb20 msgr2=0x7fc99c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 1 --2- 10.20.193.51:0/4056165522 >> v2:10.20.193.68:3300/0 conn(0x7fc99c0cdb20 0x7fc99c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc0994b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbc040cdb20 0x7fbc0415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 1 -- 10.20.193.51:0/4056165522 >> v1:10.20.193.51:6789/0 conn(0x7fc99c1526a0 legacy=0x7fc99c0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251193073 0 0) 0x7f2e840cbc80 con 0x7f2e84155e60 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 10 monclient(hunting): my global_id is 4407 2026-02-22T21:37:38.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e70003620 con 0x7f2e84155e60 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fc99affd640 1 -- 10.20.193.51:0/4056165522 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c073d60 con 0x7fc99c152a80 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10181526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1018152a80 0x7f10180cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-a con 0x7f1018152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f10180cdb20 0x7f101815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-b con 0x7f10180cdb20 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1018073ae0 con 0x7f10181526a0 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f10180cca00 con 0x7f1018152a80 2026-02-22T21:37:38.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbc04152a80 0x7fbc040cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35232/0 (socket says 10.20.193.51:35232) 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 1 -- 10.20.193.51:0/302272350 learned_addr learned my addr 10.20.193.51:0/302272350 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f1018074c50 con 0x7f10180cdb20 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f101ef11640 10 monclient(hunting): authenticate will time out at 1133.791776s 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 10 monclient(hunting): get_auth_request con 0x7fbc04152a80 auth_method 0 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7fbc08949640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.211+0000 7f73db7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f73dc152cc0 0x7f73dc0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f73dc152cc0 0x7f73dc0cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35236/0 (socket says 10.20.193.51:35236) 2026-02-22T21:37:38.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 1 -- 10.20.193.51:0/2683448957 learned_addr learned my addr 10.20.193.51:0/2683448957 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 10 monclient(hunting): get_auth_request con 0x7f73dc152cc0 auth_method 0 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73db7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.212+0000 7f73dbfff640 1 --2- 10.20.193.51:0/2683448957 >> v2:10.20.193.125:3300/0 conn(0x7f73dc15aec0 0x7f73dc158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): get_auth_request con 0x7f73dc15aec0 auth_method 0 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fc99affd640 1 --2- 10.20.193.51:0/4056165522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c152a80 0x7fc99c0cd5e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc99c15f410 tx=0x7fc98c02eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bcba856a00440d9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f7c3e7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7c4015c790 0x7f7c400c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f7c3f7fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f7c400c92a0 0x7f7c400d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46894/0 (socket says 10.20.193.51:46894) 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f7c3f7fe640 1 -- 10.20.193.51:0/1638215907 learned_addr learned my addr 10.20.193.51:0/1638215907 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc0994b640 10 monclient(hunting): get_auth_request con 0x7fbc040cdb20 auth_method 0 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc0994b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc0994b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc0994b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 10 monclient(hunting): handle_auth_done global_id 4420 payload 274 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 1 -- 10.20.193.51:0/302272350 >> v2:10.20.193.51:3300/0 conn(0x7fbc040cdb20 msgr2=0x7fbc0415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 1 --2- 10.20.193.51:0/302272350 >> v2:10.20.193.51:3300/0 conn(0x7fbc040cdb20 0x7fbc0415d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 1 -- 10.20.193.51:0/302272350 >> v1:10.20.193.51:6789/0 conn(0x7fbc041526a0 legacy=0x7fbc040733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7fbc08949640 1 -- 10.20.193.51:0/302272350 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc04073d60 con 0x7fbc04152a80 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.213+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f7c3e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c4015c790 auth_method 0 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f7c3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f7c3e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f7c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754383302 0 0) 0x7f2e840cc730 con 0x7f2e840c9470 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 10 monclient(hunting): my global_id is 4403 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e840cbc80 con 0x7f2e840c9470 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 1 -- 10.20.193.51:0/2683448957 >> v2:10.20.193.125:3300/0 conn(0x7f73dc15aec0 msgr2=0x7f73dc158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 1 --2- 10.20.193.51:0/2683448957 >> v2:10.20.193.125:3300/0 conn(0x7f73dc15aec0 0x7f73dc158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 1 -- 10.20.193.51:0/2683448957 >> v1:10.20.193.125:6789/0 conn(0x7f73dc0c58b0 legacy=0x7f73dc15a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.214+0000 7f73db7fe640 1 -- 10.20.193.51:0/2683448957 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc0c7fe0 con 0x7f73dc152cc0 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7fc1777fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fc17806d710 0x7fc1780c7100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:32998/0 (socket says 10.20.193.51:32998) 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7fc1777fe640 1 -- 10.20.193.51:0/3983043201 learned_addr learned my addr 10.20.193.51:0/3983043201 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.215+0000 7fc176ffd640 1 --2- 10.20.193.51:0/3983043201 >> v2:10.20.193.125:3300/0 conn(0x7fc178153970 0x7fc178153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7fc1767fc640 1 --2- 10.20.193.51:0/3983043201 >> v2:10.20.193.68:3300/0 conn(0x7fc17815a420 0x7fc17806f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7fd543469640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7fd543469640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7fd543469640 10 monclient: monmap: 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.217092+0000 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.217092+0000 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.259 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7fbc08949640 1 --2- 10.20.193.51:0/302272350 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc04152a80 0x7fbc040cd5e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc0415f410 tx=0x7fbbf802ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3831e78347e5f374 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.216+0000 7f73db7fe640 1 --2- 10.20.193.51:0/2683448957 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc0cefa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73dc071fe0 tx=0x7f73c802ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d41410a71255af4d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 Processor -- start 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fbe1295c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fbe1295c640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fbe1295c640 10 monclient: monmap: 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.217821+0000 2026-02-22T21:37:38.260 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.217821+0000 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): get_auth_request con 0x7fc178153970 auth_method 0 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 -- start start 2026-02-22T21:37:38.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient: init 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc1767fc640 10 monclient(hunting): get_auth_request con 0x7fc17815a420 auth_method 0 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc1767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc1767fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc1767fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7f101d487640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f10180cdb20 0x7f101815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient: _add_conns ranks=[2,4,1,5,0,3] 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd53c152790 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd53c152b70 0x7fd53c0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): picked mon.noname-b con 0x7fd53c152b70 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fd53c0cc2f0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c0cc8b0 con 0x7fd53c0cc2f0 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c0ccb90 con 0x7fd53c152790 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd53c0c9480 con 0x7fd53c152b70 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fd543469640 10 monclient(hunting): authenticate will time out at 1133.797911s 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fbe1295c640 1 Processor -- start 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.217+0000 7fbe1295c640 1 -- start start 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient: init 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f10180cdb20 0x7f101815d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35258/0 (socket says 10.20.193.51:35258) 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 1 -- 10.20.193.51:0/238896190 learned_addr learned my addr 10.20.193.51:0/238896190 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 10 monclient(hunting): get_auth_request con 0x7f10180cdb20 auth_method 0 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f101d487640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2026-02-22T21:37:38.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 1 -- 10.20.193.51:0/3983043201 >> v2:10.20.193.68:3300/0 conn(0x7fc17815a420 msgr2=0x7fc17806f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_done global_id 4413 payload 274 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 1 -- 10.20.193.51:0/1638215907 >> v1:10.20.193.68:6789/0 conn(0x7f7c40155e60 legacy=0x7f7c400cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 1 --2- 10.20.193.51:0/3983043201 >> v2:10.20.193.68:3300/0 conn(0x7fc17815a420 0x7fc17806f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 1 -- 10.20.193.51:0/1638215907 >> v1:10.20.193.51:6789/0 conn(0x7f7c400c92a0 legacy=0x7f7c400d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 1 -- 10.20.193.51:0/3983043201 >> v1:10.20.193.125:6789/0 conn(0x7fc17806d710 legacy=0x7fc1780c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7f7c3e7fc640 1 -- 10.20.193.51:0/1638215907 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c400cc450 con 0x7f7c4015c790 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fc176ffd640 1 -- 10.20.193.51:0/3983043201 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1780c5020 con 0x7fc178153970 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 1 -- 10.20.193.51:0/3534233387 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f403c070 con 0x7ff600150220 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: dump: 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.266 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient: _add_conns ranks=[3,2,4,0,5,1] 2026-02-22T21:37:38.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe0c1526a0 0x7fbe0c0733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 1 -- 10.20.193.51:0/3534233387 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f402fc10 con 0x7ff600150220 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c1526a0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe0c152a80 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff605355640 5 monclient: authenticate success, global_id 4414 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbe0c15d3b0 0x7fbe0c15d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c15d3b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 1 -- 10.20.193.51:0/3534233387 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4037440 con 0x7ff600150220 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff5fdffb640 10 monclient: dump: 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.268 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0c073d60 con 0x7fbe0c152a80 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0c0cca00 con 0x7fbe0c1526a0 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0c074c50 con 0x7fbe0c15d3b0 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7fbe1295c640 10 monclient(hunting): authenticate will time out at 1133.798914s 2026-02-22T21:37:38.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff605355640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.218+0000 7ff605355640 4 set_mon_vals no callback set 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 10 monclient: shutdown 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 -- 10.20.193.51:0/3534233387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff600150220 msgr2=0x7ff6000c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 --2- 10.20.193.51:0/3534233387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff600150220 0x7ff6000c73a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40098a0 tx=0x7ff5f402ef00 comp rx=0 tx=0).stop 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 -- 10.20.193.51:0/3534233387 shutdown_connections 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 --2- 10.20.193.51:0/3534233387 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff600150220 0x7ff6000c73a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 -- 10.20.193.51:0/3534233387 >> 10.20.193.51:0/3534233387 conn(0x7ff6000cd7b0 msgr2=0x7ff6000cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 -- 10.20.193.51:0/3534233387 shutdown_connections 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7ff605355640 1 -- 10.20.193.51:0/3534233387 wait complete. 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7f2feb986640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7f2feb986640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.219+0000 7f2feb986640 10 monclient: monmap: 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.220607+0000 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.220607+0000 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.270 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 10 monclient: monmap: 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f2feb986640 1 Processor -- start 2026-02-22T21:37:38.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f2feb986640 1 -- start start 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f2feb986640 10 monclient: init 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 librados: starting msgr at 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 librados: starting objecter 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 Processor -- start 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 1 -- 10.20.193.51:0/4056165522 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c03c070 con 0x7fc99c152a80 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: dump: 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 -- start start 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 librados: setting wanted keys 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 1 librados: calling monclient init 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 10 monclient: init 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 1 -- 10.20.193.51:0/4056165522 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c02fb80 con 0x7fc99c152a80 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc9a1d85640 5 monclient: authenticate success, global_id 4400 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc9a1d85640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc9a1d85640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 1 -- 10.20.193.51:0/4056165522 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c037440 con 0x7fc99c152a80 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc9a1d85640 4 set_mon_vals no callback set 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7fc99a7fc640 10 monclient: dump: 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.273 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.220+0000 7ff605355640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff600150220 0x7ff6001f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): picked mon.noname-c con 0x7ff600150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 0x7ff60006d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): picked mon.noname-a con 0x7ff600150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff60015c710 0x7ff60006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): picked mon.noname-b con 0x7ff60015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff6000703b0 con 0x7ff600150600 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff60015bae0 con 0x7ff60015c710 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff600070530 con 0x7ff600150220 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7ff605355640 10 monclient(hunting): authenticate will time out at 1133.801086s 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 10 monclient: shutdown 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 -- 10.20.193.51:0/4056165522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c152a80 msgr2=0x7fc99c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/4056165522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c152a80 0x7fc99c0cd5e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc99c15f410 tx=0x7fc98c02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient: _add_conns ranks=[0,1,2,3,4,5] 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fd5411de640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fd53c152790 0x7fd53c0c8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33008/0 (socket says 10.20.193.51:33008) 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fd5411de640 1 -- 10.20.193.51:0/1532826940 learned_addr learned my addr 10.20.193.51:0/1532826940 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 -- 10.20.193.51:0/4056165522 shutdown_connections 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/4056165522 >> v2:10.20.193.68:3300/0 conn(0x7fc99c0cdb20 0x7fc99c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/4056165522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c152a80 0x7fc99c0cd5e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 -- 10.20.193.51:0/4056165522 >> 10.20.193.51:0/4056165522 conn(0x7fc99c0ce350 msgr2=0x7fc99c0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 -- 10.20.193.51:0/4056165522 shutdown_connections 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fe41526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f7c3e7fc640 1 --2- 10.20.193.51:0/1638215907 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c4015c790 0x7f7c400c8d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c280098a0 tx=0x7f7c2802ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70db348ccb4313c4 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc176ffd640 1 --2- 10.20.193.51:0/3983043201 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc178153970 0x7fc178153d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc178069cb0 tx=0x7fc16c02eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfbc473764e1621e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 1 -- 10.20.193.51:0/4056165522 wait complete. 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f2fe4152a80 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2fe415d3b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe4073ae0 con 0x7f2fe41526a0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe4073d60 con 0x7f2fe4152a80 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe4074040 con 0x7f2fe415d3b0 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7f2feb986640 10 monclient(hunting): authenticate will time out at 1133.801410s 2026-02-22T21:37:38.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe0c1526a0 0x7fbe0c0733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fc9a1d85640 10 monclient: monmap: 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe0c1526a0 0x7fbe0c0733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60690/0 (socket says 10.20.193.51:60690) 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 1 -- 10.20.193.51:0/3532418539 learned_addr learned my addr 10.20.193.51:0/3532418539 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 10 monclient(hunting): get_auth_request con 0x7fbe0c1526a0 auth_method 0 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.221+0000 7fbe0bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fbe0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fbe0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 librados: starting msgr at 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fbe0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 librados: starting objecter 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f100ffff640 1 --2- 10.20.193.51:0/238896190 >> v2:10.20.193.51:3300/0 conn(0x7f1018152a80 0x7f10180cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 Processor -- start 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 -- start start 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 librados: setting wanted keys 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 librados: calling monclient init 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient: init 2026-02-22T21:37:38.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f100ffff640 10 monclient(hunting): get_auth_request con 0x7f1018152a80 auth_method 0 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f100ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f100ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f100ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 10 monclient(hunting): handle_auth_done global_id 4426 payload 274 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 1 -- 10.20.193.51:0/238896190 >> v2:10.20.193.51:3300/0 conn(0x7f1018152a80 msgr2=0x7f10180cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 1 --2- 10.20.193.51:0/238896190 >> v2:10.20.193.51:3300/0 conn(0x7f1018152a80 0x7f10180cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 1 -- 10.20.193.51:0/238896190 >> v1:10.20.193.125:6789/0 conn(0x7f10181526a0 legacy=0x7f10180733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7f101d487640 1 -- 10.20.193.51:0/238896190 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1018073d60 con 0x7f10180cdb20 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 0x7fc99c1f7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c0cdb20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc99c1526a0 0x7fc99c06b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-b con 0x7fc99c1526a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc99c152a80 0x7fc99c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): picked mon.noname-a con 0x7fc99c152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c074600 con 0x7fc99c152a80 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c074320 con 0x7fc99c1526a0 2026-02-22T21:37:38.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c0748e0 con 0x7fc99c0cdb20 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.222+0000 7fc9a1d85640 10 monclient(hunting): authenticate will time out at 1133.802841s 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 10 monclient: monmap: 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.223797+0000 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.223797+0000 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 1 Processor -- start 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 1 -- start start 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7fd59231b640 10 monclient: init 2026-02-22T21:37:38.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7f6a69d54640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7f6a69d54640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.223+0000 7f6a69d54640 10 monclient: monmap: 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.224467+0000 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.224467+0000 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 1 -- 10.20.193.51:0/3532418539 >> v2:10.20.193.68:3300/0 conn(0x7fbe0c15d3b0 msgr2=0x7fbe0c15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 1 --2- 10.20.193.51:0/3532418539 >> v2:10.20.193.68:3300/0 conn(0x7fbe0c15d3b0 0x7fbe0c15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 1 -- 10.20.193.51:0/3532418539 >> v1:10.20.193.125:6789/0 conn(0x7fbe0c152a80 legacy=0x7fbe0c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fbe0bfff640 1 -- 10.20.193.51:0/3532418539 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c074040 con 0x7fbe0c1526a0 2026-02-22T21:37:38.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient: _add_conns ranks=[0,5,3,1,2,4] 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 Processor -- start 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 1 -- 10.20.193.51:0/2683448957 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c803c070 con 0x7f73dc152cc0 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: dump: 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 1 -- 10.20.193.51:0/2683448957 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c802fa90 con 0x7f73dc152cc0 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 1 -- 10.20.193.51:0/2683448957 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c802fe60 con 0x7f73dc152cc0 2026-02-22T21:37:38.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 5 monclient: authenticate success, global_id 4423 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 4 set_mon_vals no callback set 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73da7fc640 10 monclient: dump: 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd58c1526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd58c152a80 0x7fd58c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c152a80 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd58c0cdb20 0x7fd58c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 10 monclient: monmap: 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.224996+0000 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.224996+0000 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.284 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c0cdb20 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 -- start start 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient: init 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58c073ae0 con 0x7fd58c1526a0 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fd58c0cca00 con 0x7fd58c0cdb20 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd58c074c50 con 0x7fd58c152a80 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7fd59231b640 10 monclient(hunting): authenticate will time out at 1133.804467s 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f101d487640 1 --2- 10.20.193.51:0/238896190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10180cdb20 0x7f101815d3b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f101808f0d0 tx=0x7f101402f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c241c54fcf20add server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 0x7ff60006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 10 monclient: shutdown 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 -- 10.20.193.51:0/2683448957 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 msgr2=0x7f73dc0cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/2683448957 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc0cefa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73dc071fe0 tx=0x7f73c802ee60 comp rx=0 tx=0).stop 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 0x7ff60006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60700/0 (socket says 10.20.193.51:60700) 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 1 -- 10.20.193.51:0/3356402320 learned_addr learned my addr 10.20.193.51:0/3356402320 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 10 monclient(hunting): get_auth_request con 0x7ff600150600 auth_method 0 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7ff5fe7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient: _add_conns ranks=[4,3,1,2,0,5] 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6a64156010 0x7f6a640cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-b con 0x7f6a64156010 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 1 Processor -- start 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6a640caca0 0x7f6a640c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-a con 0x7f6a640caca0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f6a640c9020 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a640c9920 con 0x7f6a640c9020 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f6a640c96d0 con 0x7f6a640caca0 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f6a640ca810 con 0x7f6a64156010 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f6a69d54640 10 monclient(hunting): authenticate will time out at 1133.804885s 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 -- 10.20.193.51:0/2683448957 shutdown_connections 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/2683448957 >> v2:10.20.193.125:3300/0 conn(0x7f73dc15aec0 0x7f73dc158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/2683448957 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc0cefa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 -- 10.20.193.51:0/2683448957 >> 10.20.193.51:0/2683448957 conn(0x7f73dc0ce4b0 msgr2=0x7f73dc0ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f4ccbb96640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f4ccbb96640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f4ccbb96640 10 monclient: monmap: 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.225525+0000 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.225525+0000 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.287 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 1 -- start start 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f2841b40640 10 monclient: init 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 -- 10.20.193.51:0/2683448957 shutdown_connections 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.224+0000 7f73e1f5c640 1 -- 10.20.193.51:0/2683448957 wait complete. 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7ff5fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7ff5fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7ff5fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 Processor -- start 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 -- start start 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient: init 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 10 monclient: monmap: 2026-02-22T21:37:38.288 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient: _add_conns ranks=[4,2,3,0,1,5] 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 0x7f283c15fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-b con 0x7f283c155ec0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f283c0ccba0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f283c15da20 0x7f283c0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-a con 0x7f283c15da20 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f283c160360 con 0x7f283c0ccba0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f283c06f520 con 0x7f283c15da20 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f283c0d0510 con 0x7f283c155ec0 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f2841b40640 10 monclient(hunting): authenticate will time out at 1133.805514s 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 librados: starting msgr at 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 librados: starting objecter 2026-02-22T21:37:38.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient: _add_conns ranks=[4,2,1,5,3,0] 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc4152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 Processor -- start 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4cc406d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f4cc40c95a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc406f1d0 con 0x7f4cc40c95a0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc406de50 con 0x7f4cc406d9d0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc406ed40 con 0x7f4cc4152ab0 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 -- start start 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 librados: setting wanted keys 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f4ccbb96640 10 monclient(hunting): authenticate will time out at 1133.805914s 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 1 librados: calling monclient init 2026-02-22T21:37:38.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.225+0000 7f73e1f5c640 10 monclient: init 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7fb6833ca640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7fb6833ca640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7fb6833ca640 10 monclient: monmap: 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.226746+0000 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.226746+0000 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1447875406 0 0) 0x7f2e840cbc80 con 0x7f2e840c9470 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 >> v1:10.20.193.68:6789/0 conn(0x7f2e8415c790 legacy=0x7f2e840c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 >> v1:10.20.193.51:6789/0 conn(0x7f2e84155e60 legacy=0x7f2e840cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e840cca10 con 0x7f2e840c9470 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f2e74003260 con 0x7f2e840c9470 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: dump: 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 >> v1:10.20.193.125:6789/0 conn(0x7f2e840c9470 legacy=0x7f2e840d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e70003dd0 0x7f2e700041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): picked mon.a con 0x7f2e70003dd0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70004900 0x7f2e70004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): picked mon.b con 0x7f2e70004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 0x7f2e7000bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): picked mon.c con 0x7f2e70009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7000c4c0 con 0x7f2e70003dd0 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7000c670 con 0x7f2e70004900 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7000c870 con 0x7f2e70009ab0 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f2e81ffb640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f73dc0c58b0 0x7f73dc1f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc0c58b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-b con 0x7f73dc152cc0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 0x7f73dc145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): picked mon.noname-a con 0x7f73dc15aec0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc15a290 con 0x7f73dc15aec0 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc0c85a0 con 0x7f73dc152cc0 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc0c8880 con 0x7f73dc0c58b0 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7f73e1f5c640 10 monclient(hunting): authenticate will time out at 1133.806462s 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.226+0000 7fb6833ca640 1 Processor -- start 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 1 -- 10.20.193.51:0/302272350 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf803c070 con 0x7fbc04152a80 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: dump: 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.293 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 1 -- 10.20.193.51:0/302272350 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf802fbd0 con 0x7fbc04152a80 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 1 -- 10.20.193.51:0/302272350 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8037440 con 0x7fbc04152a80 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 5 monclient: authenticate success, global_id 4420 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 4 set_mon_vals no callback set 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 1 -- start start 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient: init 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbbf3fff640 10 monclient: dump: 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 1 -- 10.20.193.51:0/1532826940 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716759610 0 0) 0x7fd53c0ccb90 con 0x7fd53c152790 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 10 monclient(hunting): my global_id is 4415 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fd52bfff640 1 -- 10.20.193.51:0/1532826940 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd524003620 con 0x7fd53c152790 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 10 monclient: shutdown 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/302272350 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc04152a80 msgr2=0x7fbc040cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/302272350 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc04152a80 0x7fbc040cd5e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc0415f410 tx=0x7fbbf802ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient: _add_conns ranks=[1,5,2,4,0,3] 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb67c1526a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb67c152a80 0x7fb67c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c152a80 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb67c0cdb20 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c073ae0 con 0x7fb67c1526a0 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c073d60 con 0x7fb67c0cdb20 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fb67c0cca00 con 0x7fb67c152a80 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 -- 10.20.193.51:0/3356402320 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff600150220 msgr2=0x7ff6001f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff600150220 0x7ff6001f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fb6833ca640 10 monclient(hunting): authenticate will time out at 1133.807641s 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 -- 10.20.193.51:0/3356402320 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff60015c710 msgr2=0x7ff60006d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff60015c710 0x7ff60006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7ff5fe7fc640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60014b190 con 0x7ff600150600 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/302272350 shutdown_connections 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/302272350 >> v2:10.20.193.51:3300/0 conn(0x7fbc040cdb20 0x7fbc0415d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/302272350 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc04152a80 0x7fbc040cd5e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/302272350 >> 10.20.193.51:0/302272350 conn(0x7fbc040ce350 msgr2=0x7fbc040ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/302272350 shutdown_connections 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.227+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/302272350 wait complete. 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1532826940 >> v2:10.20.193.68:3300/0 conn(0x7fd53c152b70 0x7fd53c0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 10 monclient: monmap: 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 librados: starting msgr at 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 librados: starting objecter 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 Processor -- start 2026-02-22T21:37:38.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fc99b7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 0x7fc99c1f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 -- start start 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 librados: setting wanted keys 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 1 librados: calling monclient init 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fbc0b3d5640 10 monclient: init 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd58c0cdb20 0x7fd58c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd5409dd640 10 monclient(hunting): get_auth_request con 0x7fd53c152b70 auth_method 0 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd5409dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd5409dd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd5409dd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd58c0cdb20 0x7fd58c15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60718/0 (socket says 10.20.193.51:60718) 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 1 -- 10.20.193.51:0/2411899773 learned_addr learned my addr 10.20.193.51:0/2411899773 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 10 monclient(hunting): get_auth_request con 0x7fd58c0cdb20 auth_method 0 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.228+0000 7fd590891640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc040cdb20 0x7fbc041481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-b con 0x7fbc040cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 0x7fbc0414b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-c con 0x7fbc041526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc04152a80 0x7fbc0414b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): picked mon.noname-a con 0x7fbc04152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbc04074600 con 0x7fbc04152a80 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbc04074320 con 0x7fbc040cdb20 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040748e0 con 0x7fbc041526a0 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fbc0b3d5640 10 monclient(hunting): authenticate will time out at 1133.809109s 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd590891640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd590891640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fd590891640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 0x7fc99c1f7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43614/0 (socket says 10.20.193.51:43614) 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 1 -- 10.20.193.51:0/225000125 learned_addr learned my addr 10.20.193.51:0/225000125 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 10 monclient(hunting): get_auth_request con 0x7fc99c0cdb20 auth_method 0 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.229+0000 7fc99b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 10 monclient(hunting): handle_auth_done global_id 4432 payload 274 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 1 -- 10.20.193.51:0/1532826940 >> v1:10.20.193.125:6789/0 conn(0x7fd53c152790 legacy=0x7fd53c0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 1 -- 10.20.193.51:0/1532826940 >> v1:10.20.193.68:6789/0 conn(0x7fd53c0cc2f0 legacy=0x7fd53c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 1 -- 10.20.193.51:0/2411899773 >> v2:10.20.193.125:3300/0 conn(0x7fd58c152a80 msgr2=0x7fd58c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 1 --2- 10.20.193.51:0/2411899773 >> v2:10.20.193.125:3300/0 conn(0x7fd58c152a80 0x7fd58c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 1 -- 10.20.193.51:0/2411899773 >> v1:10.20.193.51:6789/0 conn(0x7fd58c1526a0 legacy=0x7fd58c0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd590891640 1 -- 10.20.193.51:0/2411899773 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c073d60 con 0x7fd58c0cdb20 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.230+0000 7fd5409dd640 1 -- 10.20.193.51:0/1532826940 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c0cce70 con 0x7fd53c152b70 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.231+0000 7fd590891640 1 --2- 10.20.193.51:0/2411899773 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c0cdb20 0x7fd58c15d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd58c15f410 tx=0x7fd58002f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd06f8de3561a17f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.232+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1532826940 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152b70 0x7fd53c0ce380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098a0 tx=0x7fd53002ee10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548edbde8bcfc726 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.232+0000 7ff5fe7fc640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 0x7ff60006d170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c4d0 tx=0x7ff5e800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.232+0000 7ff5df7fe640 10 client.?.objecter ms_handle_connect 0x7ff600150600 2026-02-22T21:37:38.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.232+0000 7ff5df7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.232+0000 7fc99affd640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc99c1526a0 0x7fc99c06b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99bfff640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc99c152a80 0x7fc99c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 -- 10.20.193.51:0/225000125 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc99c1526a0 msgr2=0x7fc99c06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc99c1526a0 0x7fc99c06b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 -- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc99c152a80 msgr2=0x7fc99c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc99c152a80 0x7fc99c15fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7fc99b7fe640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c15d730 con 0x7fc99c0cdb20 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7f2e82ffd640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e70003dd0 0x7f2e700041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.233+0000 7f2e837fe640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 0x7f2e7000bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): get_auth_request con 0x7f2e70009ab0 auth_method 0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6a64156010 0x7f6a640cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f2e837fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a63fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6a640caca0 0x7f6a640c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6a64156010 0x7f6a640cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35304/0 (socket says 10.20.193.51:35304) 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 1 -- 10.20.193.51:0/2956282006 learned_addr learned my addr 10.20.193.51:0/2956282006 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 10 monclient(hunting): get_auth_request con 0x7f6a64156010 auth_method 0 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.234+0000 7f6a68d52640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 0x7f283c15fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7fc99b7fe640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 0x7fc99c1f7850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc98800c700 tx=0x7fc98800cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7fc977fff640 10 client.?.objecter ms_handle_connect 0x7fc99c0cdb20 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7fc977fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 0x7f283c15fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35306/0 (socket says 10.20.193.51:35306) 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 1 -- 10.20.193.51:0/1883859510 learned_addr learned my addr 10.20.193.51:0/1883859510 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 10 monclient(hunting): get_auth_request con 0x7f283c155ec0 auth_method 0 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f283b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f2e82ffd640 10 monclient(hunting): get_auth_request con 0x7f2e70003dd0 auth_method 0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f2e82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f2e82ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f2e82ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.235+0000 7f2e827fc640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70004900 0x7f2e70004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 10 monclient(hunting): handle_auth_done global_id 4403 payload 293 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 -- 10.20.193.51:0/63750042 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70004900 msgr2=0x7f2e70004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70004900 0x7f2e70004d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 -- 10.20.193.51:0/63750042 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e70003dd0 msgr2=0x7f2e700041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e70003dd0 0x7f2e700041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 monclient: found mon.c 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 -- 10.20.193.51:0/63750042 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e840cbc80 con 0x7f2e70009ab0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e897ce640 5 monclient: authenticate success, global_id 4403 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e897ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f2e837fe640 1 -- 10.20.193.51:0/63750042 reap_dead start 2026-02-22T21:37:38.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a63fff640 10 monclient(hunting): get_auth_request con 0x7f6a640caca0 auth_method 0 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a63fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a63fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 10 monclient(hunting): handle_auth_done global_id 4438 payload 274 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 1 -- 10.20.193.51:0/2956282006 >> v2:10.20.193.51:3300/0 conn(0x7f6a640caca0 msgr2=0x7f6a640c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 1 --2- 10.20.193.51:0/2956282006 >> v2:10.20.193.51:3300/0 conn(0x7f6a640caca0 0x7f6a640c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 1 -- 10.20.193.51:0/2956282006 >> v1:10.20.193.68:6789/0 conn(0x7f6a640c9020 legacy=0x7f6a6415bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.236+0000 7f6a68d52640 1 -- 10.20.193.51:0/2956282006 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a640c9c00 con 0x7f6a64156010 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 1 -- 10.20.193.51:0/238896190 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101402fd70 con 0x7f10180cdb20 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: dump: 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.303 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 1 -- 10.20.193.51:0/238896190 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10140373f0 con 0x7f10180cdb20 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 5 monclient: authenticate success, global_id 4426 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 4 set_mon_vals no callback set 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 1 -- 10.20.193.51:0/238896190 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140377c0 con 0x7f10180cdb20 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f100f7fe640 10 monclient: dump: 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 10 monclient: shutdown 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 -- 10.20.193.51:0/238896190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10180cdb20 msgr2=0x7f101815d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 --2- 10.20.193.51:0/238896190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10180cdb20 0x7f101815d3b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f101808f0d0 tx=0x7f101402f310 comp rx=0 tx=0).stop 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 -- 10.20.193.51:0/238896190 shutdown_connections 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 --2- 10.20.193.51:0/238896190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10180cdb20 0x7f101815d3b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 --2- 10.20.193.51:0/238896190 >> v2:10.20.193.51:3300/0 conn(0x7f1018152a80 0x7f10180cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 -- 10.20.193.51:0/238896190 >> 10.20.193.51:0/238896190 conn(0x7f10180ce350 msgr2=0x7f10180ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 -- 10.20.193.51:0/238896190 shutdown_connections 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.237+0000 7f101ef11640 1 -- 10.20.193.51:0/238896190 wait complete. 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f283bfff640 1 --2- 10.20.193.51:0/1883859510 >> v2:10.20.193.51:3300/0 conn(0x7f283c15da20 0x7f283c0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f2e837fe640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 0x7f2e7000bea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400a730 tx=0x7f2e7402fb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9965b8dbe5026c7c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 10 monclient: monmap: 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35320/0 (socket says 10.20.193.51:35320) 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 1 -- 10.20.193.51:0/1813198351 learned_addr learned my addr 10.20.193.51:0/1813198351 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 10 monclient(hunting): get_auth_request con 0x7f4cc4152ab0 auth_method 0 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f4cc990b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f6a68d52640 1 --2- 10.20.193.51:0/2956282006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640cc060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a6415f7b0 tx=0x7f6a5402eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fded5e33ddd5ada2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 librados: starting msgr at 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 librados: starting objecter 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 Processor -- start 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 0x7f73dc145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 -- start start 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 librados: setting wanted keys 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 1 librados: calling monclient init 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.238+0000 7f101ef11640 10 monclient: init 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f4cc990b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f4cc990b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f4cc990b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283bfff640 10 monclient(hunting): get_auth_request con 0x7f283c15da20 auth_method 0 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f283bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 0x7f73dc145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60746/0 (socket says 10.20.193.51:60746) 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 1 -- 10.20.193.51:0/984994084 learned_addr learned my addr 10.20.193.51:0/984994084 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 10 monclient(hunting): get_auth_request con 0x7f73dc15aec0 auth_method 0 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f10180cdb20 0x7f1018144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-a con 0x7f10180cdb20 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f10181526a0 0x7f10181474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73dbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-c con 0x7f10181526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1018152a80 0x7f1018147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): picked mon.noname-b con 0x7f1018152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1018074600 con 0x7f10180cdb20 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1018074320 con 0x7f1018152a80 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f10180748e0 con 0x7f10181526a0 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f101ef11640 10 monclient(hunting): authenticate will time out at 1133.819551s 2026-02-22T21:37:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7f73daffd640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.239+0000 7fb68093e640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb67c152a80 0x7fb67c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f283bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f283bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f283bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 10 monclient(hunting): handle_auth_done global_id 4441 payload 274 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 1 -- 10.20.193.51:0/1813198351 >> v1:10.20.193.125:6789/0 conn(0x7f4cc406d9d0 legacy=0x7f4cc40c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 1 -- 10.20.193.51:0/1813198351 >> v1:10.20.193.68:6789/0 conn(0x7f4cc40c95a0 legacy=0x7f4cc4158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.240+0000 7f4cc990b640 1 -- 10.20.193.51:0/1813198351 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc406e130 con 0x7f4cc4152ab0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb67c152a80 0x7fb67c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43640/0 (socket says 10.20.193.51:43640) 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 1 -- 10.20.193.51:0/2046266962 learned_addr learned my addr 10.20.193.51:0/2046266962 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): get_auth_request con 0x7fb67c152a80 auth_method 0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f73db7fe640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f73dc0c58b0 0x7f73dc1f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f73daffd640 10 monclient(hunting): get_auth_request con 0x7f73dc152cc0 auth_method 0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f73daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f73daffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f73daffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 10 monclient(hunting): handle_auth_done global_id 4434 payload 274 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 1 -- 10.20.193.51:0/1883859510 >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 msgr2=0x7f283c15fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 1 --2- 10.20.193.51:0/1883859510 >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 0x7f283c15fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 1 -- 10.20.193.51:0/1883859510 >> v1:10.20.193.125:6789/0 conn(0x7f283c0ccba0 legacy=0x7f283c15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7f283bfff640 1 -- 10.20.193.51:0/1883859510 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c06f770 con 0x7f283c15da20 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 1 -- 10.20.193.51:0/3983043201 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c03c070 con 0x7fc178153970 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: dump: 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 1 -- 10.20.193.51:0/3983043201 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c02fb80 con 0x7fc178153970 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 1 -- 10.20.193.51:0/3983043201 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c037440 con 0x7fc178153970 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc175ffb640 10 monclient: dump: 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc17d944640 5 monclient: authenticate success, global_id 4409 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc17d944640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fc17d944640 4 set_mon_vals no callback set 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.241+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fbc08949640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 0x7fbc0414b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7f73db7fe640 10 monclient(hunting): get_auth_request con 0x7f73dc0c58b0 auth_method 0 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7f73db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7f73db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7f73db7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 10 monclient: shutdown 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 -- 10.20.193.51:0/3983043201 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc178153970 msgr2=0x7fc178153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 --2- 10.20.193.51:0/3983043201 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc178153970 0x7fc178153d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc178069cb0 tx=0x7fc16c02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 -- 10.20.193.51:0/3983043201 shutdown_connections 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 --2- 10.20.193.51:0/3983043201 >> v2:10.20.193.68:3300/0 conn(0x7fc17815a420 0x7fc17806f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 --2- 10.20.193.51:0/3983043201 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc178153970 0x7fc178153d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 -- 10.20.193.51:0/3983043201 >> 10.20.193.51:0/3983043201 conn(0x7fc1780ca240 msgr2=0x7fc1780ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 -- 10.20.193.51:0/3983043201 shutdown_connections 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fc17d944640 1 -- 10.20.193.51:0/3983043201 wait complete. 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 1 -- 10.20.193.51:0/2046266962 >> v1:10.20.193.125:6789/0 conn(0x7fb67c0cdb20 legacy=0x7fb67c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 1 -- 10.20.193.51:0/2046266962 >> v1:10.20.193.68:6789/0 conn(0x7fb67c1526a0 legacy=0x7fb67c0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.242+0000 7fb68093e640 1 -- 10.20.193.51:0/2046266962 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c074040 con 0x7fb67c152a80 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 0x7fbc0414b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43644/0 (socket says 10.20.193.51:43644) 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 1 -- 10.20.193.51:0/4034623388 learned_addr learned my addr 10.20.193.51:0/4034623388 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): get_auth_request con 0x7fbc041526a0 auth_method 0 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient: monmap: 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 -- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f73dc0c58b0 msgr2=0x7f73dc1f7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f73dc0c58b0 0x7f73dc1f7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 -- 10.20.193.51:0/984994084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 msgr2=0x7f73dc145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f73dc152cc0 0x7f73dc145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7f73dbfff640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc0ccf60 con 0x7f73dc15aec0 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 librados: starting msgr at 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 librados: starting objecter 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 Processor -- start 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 -- start start 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 librados: setting wanted keys 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 librados: calling monclient init 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient: init 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fbc08949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc17806d710 0x7fc178073700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-a con 0x7fc17806d710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc178153970 0x7fc1780bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-b con 0x7fc178153970 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc17815a420 0x7fc17814f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): picked mon.noname-c con 0x7fc17815a420 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc17815a890 con 0x7fc17806d710 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc1780c55e0 con 0x7fc178153970 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc178070cc0 con 0x7fc17815a420 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.243+0000 7fc17d944640 10 monclient(hunting): authenticate will time out at 1133.823943s 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.244+0000 7f4cc990b640 1 --2- 10.20.193.51:0/1813198351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400bcc0 tx=0x7f4cb40317f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65784b6be35f7fe6 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.244+0000 7fbe0bfff640 1 --2- 10.20.193.51:0/3532418539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 0x7fbe0c0733d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c08f0d0 tx=0x7fbdf802ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1799cee88793245 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.244+0000 7f2fe9efc640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f2fe415d3b0 0x7f2fe415f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33030/0 (socket says 10.20.193.51:33030) 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.244+0000 7f2fe9efc640 1 -- 10.20.193.51:0/1907496938 learned_addr learned my addr 10.20.193.51:0/1907496938 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7f73dbfff640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 0x7f73dc145db0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73d000efc0 tx=0x7f73d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7f73b7fff640 10 client.?.objecter ms_handle_connect 0x7f73dc15aec0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7f73b7fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 -- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc040cdb20 msgr2=0x7fbc041481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc040cdb20 0x7fbc041481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 -- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc04152a80 msgr2=0x7fbc0414b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc04152a80 0x7fbc0414b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.245+0000 7fbc08949640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040d1630 con 0x7fbc041526a0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.246+0000 7fb68093e640 1 --2- 10.20.193.51:0/2046266962 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 0x7fb67c0cd5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb670009970 tx=0x7fb67002efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282ba2cb70ebddd4 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.246+0000 7f283bfff640 1 --2- 10.20.193.51:0/1883859510 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c15da20 0x7f283c0caf90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f283c0751e0 tx=0x7f283002f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7203117a889011 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.248+0000 7fbc08949640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 0x7fbc0414b1c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8002410 tx=0x7fbbf80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.248+0000 7fbbf1ffb640 10 client.?.objecter ms_handle_connect 0x7fbc041526a0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.248+0000 7fbbf1ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 1 -- 10.20.193.51:0/1532826940 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53003c070 con 0x7fd53c152b70 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: dump: 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 1 -- 10.20.193.51:0/1532826940 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd53002fad0 con 0x7fd53c152b70 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 1 -- 10.20.193.51:0/1532826940 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530037440 con 0x7fd53c152b70 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 5 monclient: authenticate success, global_id 4432 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 4 set_mon_vals no callback set 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd52bfff640 10 monclient: dump: 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 1 -- 10.20.193.51:0/2411899773 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58002fd70 con 0x7fd58c0cdb20 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: dump: 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 1 -- 10.20.193.51:0/2411899773 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5800373f0 con 0x7fd58c0cdb20 2026-02-22T21:37:38.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 1 -- 10.20.193.51:0/2411899773 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5800377c0 con 0x7fd58c0cdb20 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd58affd640 10 monclient: dump: 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 5 monclient: authenticate success, global_id 4428 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 4 set_mon_vals no callback set 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 10 monclient: shutdown 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 1 -- 10.20.193.51:0/1532826940 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152b70 msgr2=0x7fd53c0ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd543469640 1 --2- 10.20.193.51:0/1532826940 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152b70 0x7fd53c0ce380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5300098a0 tx=0x7fd53002ee10 comp rx=0 tx=0).stop 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 10 monclient: shutdown 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 1 -- 10.20.193.51:0/2411899773 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c0cdb20 msgr2=0x7fd58c15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.249+0000 7fd59231b640 1 --2- 10.20.193.51:0/2411899773 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c0cdb20 0x7fd58c15d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd58c15f410 tx=0x7fd58002f310 comp rx=0 tx=0).stop 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 -- 10.20.193.51:0/1532826940 shutdown_connections 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 --2- 10.20.193.51:0/1532826940 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152b70 0x7fd53c0ce380 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 -- 10.20.193.51:0/1532826940 >> 10.20.193.51:0/1532826940 conn(0x7fd53c06d9c0 msgr2=0x7fd53c06ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 -- 10.20.193.51:0/1532826940 shutdown_connections 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 -- 10.20.193.51:0/2411899773 shutdown_connections 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 --2- 10.20.193.51:0/2411899773 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c0cdb20 0x7fd58c15d3b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 --2- 10.20.193.51:0/2411899773 >> v2:10.20.193.125:3300/0 conn(0x7fd58c152a80 0x7fd58c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 -- 10.20.193.51:0/2411899773 >> 10.20.193.51:0/2411899773 conn(0x7fd58c0ce350 msgr2=0x7fd58c0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 -- 10.20.193.51:0/1532826940 wait complete. 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 -- 10.20.193.51:0/2411899773 shutdown_connections 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 1 -- 10.20.193.51:0/2411899773 wait complete. 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 10 monclient: monmap: 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd59231b640 10 monclient: monmap: 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.318 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 librados: starting msgr at 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 librados: starting objecter 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7f6a637fe640 1 -- 10.20.193.51:0/2956282006 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5403c070 con 0x7f6a64156010 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7f6a637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.250+0000 7fd543469640 1 Processor -- start 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 -- start start 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 librados: setting wanted keys 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 librados: calling monclient init 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient: init 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: dump: 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.319 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 1 -- 10.20.193.51:0/2956282006 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a5402fb40 con 0x7f6a64156010 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 5 monclient: authenticate success, global_id 4438 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 librados: starting msgr at 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 librados: starting objecter 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 Processor -- start 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 4 set_mon_vals no callback set 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 1 -- 10.20.193.51:0/2956282006 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54037440 con 0x7f6a64156010 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a637fe640 10 monclient: dump: 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.320 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd53c0cc2f0 0x7fd53c1f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): picked mon.noname-c con 0x7fd53c0cc2f0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152790 0x7fd53c0c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): picked mon.noname-b con 0x7fd53c152790 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd53c152b70 0x7fd53c142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): picked mon.noname-a con 0x7fd53c152b70 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c0cd430 con 0x7fd53c152b70 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c0cd7a0 con 0x7fd53c152790 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c0cdc30 con 0x7fd53c0cc2f0 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd543469640 10 monclient(hunting): authenticate will time out at 1133.831657s 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 -- start start 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 librados: setting wanted keys 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 librados: calling monclient init 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient: init 2026-02-22T21:37:38.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 10 monclient: shutdown 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 -- 10.20.193.51:0/2956282006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 msgr2=0x7f6a640cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 --2- 10.20.193.51:0/2956282006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640cc060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a6415f7b0 tx=0x7f6a5402eed0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 -- 10.20.193.51:0/2956282006 shutdown_connections 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 --2- 10.20.193.51:0/2956282006 >> v2:10.20.193.51:3300/0 conn(0x7f6a640caca0 0x7f6a640c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 --2- 10.20.193.51:0/2956282006 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640cc060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 -- 10.20.193.51:0/2956282006 >> 10.20.193.51:0/2956282006 conn(0x7f6a640ce340 msgr2=0x7f6a640ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 -- 10.20.193.51:0/2956282006 shutdown_connections 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7f6a69d54640 1 -- 10.20.193.51:0/2956282006 wait complete. 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd58c0cdb20 0x7fd58c1fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c0cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 0x7fd58c1fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd58c152a80 0x7fd58c1f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c074600 con 0x7fd58c1526a0 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c074320 con 0x7fd58c0cdb20 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c0748e0 con 0x7fd58c152a80 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.251+0000 7fd59231b640 10 monclient(hunting): authenticate will time out at 1133.831736s 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 1 -- 10.20.193.51:0/3532418539 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf803c070 con 0x7fbe0c1526a0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: dump: 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 1 -- 10.20.193.51:0/3532418539 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf802fbc0 con 0x7fbe0c1526a0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 1 -- 10.20.193.51:0/3532418539 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8037440 con 0x7fbe0c1526a0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 5 monclient: authenticate success, global_id 4419 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 4 set_mon_vals no callback set 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe0affd640 10 monclient: dump: 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.323 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient: monmap: 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f10180cdb20 0x7f1018144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988017070 con 0x7fc99c0cdb20 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: dump: 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc988005bb0 con 0x7fc99c0cdb20 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98800a640 con 0x7fc99c0cdb20 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc999ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc977fff640 10 monclient: dump: 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 5 monclient: authenticate success, global_id 4418 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 monclient: _renew_subs 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99c15da10 con 0x7fc99c0cdb20 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 librados: starting msgr at 2026-02-22T21:37:38.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 librados: starting objecter 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 monclient: _renew_subs 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99c15dfd0 con 0x7fc99c0cdb20 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 1 librados: init done 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 Processor -- start 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 -- start start 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 librados: setting wanted keys 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 librados: calling monclient init 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient: init 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650482913 0 0) 0x7f2fe4073d60 con 0x7f2fe4152a80 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 10 monclient(hunting): my global_id is 4450 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fcc003620 con 0x7f2fe4152a80 2026-02-22T21:37:38.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74038070 con 0x7f2e70009ab0 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: dump: 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 10 monclient: shutdown 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 -- 10.20.193.51:0/3532418539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 msgr2=0x7fbe0c0733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 --2- 10.20.193.51:0/3532418539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 0x7fbe0c0733d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c08f0d0 tx=0x7fbdf802ef90 comp rx=0 tx=0).stop 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e74034680 con 0x7f2e70009ab0 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e897ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 1 -- 10.20.193.51:0/63750042 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400a350 con 0x7f2e70009ab0 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e81ffb640 10 monclient: dump: 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.327 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e897ce640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 -- 10.20.193.51:0/3532418539 shutdown_connections 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 --2- 10.20.193.51:0/3532418539 >> v2:10.20.193.68:3300/0 conn(0x7fbe0c15d3b0 0x7fbe0c15d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f2e897ce640 4 set_mon_vals no callback set 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 --2- 10.20.193.51:0/3532418539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 0x7fbe0c0733d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 -- 10.20.193.51:0/3532418539 >> 10.20.193.51:0/3532418539 conn(0x7fbe0c0ce350 msgr2=0x7fbe0c0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 0x7f6a641f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 -- 10.20.193.51:0/3532418539 shutdown_connections 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-c con 0x7f6a640c9020 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a640caca0 0x7f6a641517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-a con 0x7f6a640caca0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): picked mon.noname-b con 0x7f6a64156010 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640ca4a0 con 0x7f6a640caca0 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640ca1c0 con 0x7f6a64156010 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fbe1295c640 1 -- 10.20.193.51:0/3532418539 wait complete. 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 client.4418.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 15 client.4418.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 20 client.4418.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640c9420 con 0x7f6a640c9020 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f6a69d54640 10 monclient(hunting): authenticate will time out at 1133.832906s 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc99c06a0b0 con 0x7fc99c0cdb20 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101d487640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1018152a80 0x7f1018147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f10180cdb20 0x7f1018144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60774/0 (socket says 10.20.193.51:60774) 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 1 -- 10.20.193.51:0/225282276 learned_addr learned my addr 10.20.193.51:0/225282276 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 10 monclient(hunting): get_auth_request con 0x7f10180cdb20 auth_method 0 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.252+0000 7f101cc86640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 10 monclient: shutdown 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 -- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 msgr2=0x7f2e7000bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 0x7f2e7000bea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400a730 tx=0x7f2e7402fb10 comp rx=0 tx=0).stop 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 -- 10.20.193.51:0/63750042 shutdown_connections 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 --2- 10.20.193.51:0/63750042 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e70009ab0 0x7f2e7000bea0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 -- 10.20.193.51:0/63750042 >> 10.20.193.51:0/63750042 conn(0x7f2e8406d8c0 msgr2=0x7f2e8406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 1 -- 10.20.193.51:0/1813198351 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4036710 con 0x7f4cc4152ab0 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: dump: 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 -- 10.20.193.51:0/63750042 shutdown_connections 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 1 -- 10.20.193.51:0/63750042 wait complete. 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 1 -- 10.20.193.51:0/1813198351 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb40368b0 con 0x7f4cc4152ab0 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 1 -- 10.20.193.51:0/1813198351 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4036c80 con 0x7f4cc4152ab0 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4cc8909640 10 monclient: dump: 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 5 monclient: authenticate success, global_id 4441 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 4 set_mon_vals no callback set 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fbe1295c640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): get_auth_request con 0x7f1018152a80 auth_method 0 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fbe1295c640 10 monclient: monmap: 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 10 monclient: shutdown 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 -- 10.20.193.51:0/1813198351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 msgr2=0x7f4cc40c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 --2- 10.20.193.51:0/1813198351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400bcc0 tx=0x7f4cb40317f0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 -- 10.20.193.51:0/1813198351 shutdown_connections 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 --2- 10.20.193.51:0/1813198351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc40c5ab0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 -- 10.20.193.51:0/1813198351 >> 10.20.193.51:0/1813198351 conn(0x7f4cc40cd7b0 msgr2=0x7f4cc40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101cc86640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101cc86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc1767fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc178153970 0x7fc1780bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101cc86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc17806d710 0x7fc178073700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f2e897ce640 10 monclient: monmap: 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fbe1295c640 1 librados: starting msgr at 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fbe1295c640 1 librados: starting objecter 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f101d487640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc17806d710 0x7fc178073700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60776/0 (socket says 10.20.193.51:60776) 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 1 -- 10.20.193.51:0/4121244979 learned_addr learned my addr 10.20.193.51:0/4121244979 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 10 monclient(hunting): get_auth_request con 0x7fc17806d710 auth_method 0 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fc176ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7fbe1295c640 1 Processor -- start 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 -- start start 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 librados: setting wanted keys 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 librados: calling monclient init 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient: init 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 librados: starting msgr at 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 librados: starting objecter 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fc1767fc640 10 monclient(hunting): get_auth_request con 0x7fc178153970 auth_method 0 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fc1767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fc1767fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fc1767fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 -- 10.20.193.51:0/1813198351 shutdown_connections 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.253+0000 7f4ccbb96640 1 -- 10.20.193.51:0/1813198351 wait complete. 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 Processor -- start 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 -- start start 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 librados: setting wanted keys 2026-02-22T21:37:38.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 librados: calling monclient init 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient: init 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 10 monclient: monmap: 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 0x7fbe0c0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 0x7fbe0c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe0c15d3b0 0x7fbe0c15e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c15d3b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c0748e0 con 0x7fbe0c1526a0 2026-02-22T21:37:38.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c074600 con 0x7fbe0c15d3b0 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c0cdcf0 con 0x7fbe0c152a80 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7fbe1295c640 10 monclient(hunting): authenticate will time out at 1133.834490s 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 librados: starting msgr at 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 librados: starting objecter 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70009ab0 0x7f2e8415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-b con 0x7f2e70009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 0x7f2e840d0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-a con 0x7f2e841f3700 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e840d0e60 0x7f2e84148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): picked mon.noname-c con 0x7f2e840d0e60 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2e840ccfd0 con 0x7f2e841f3700 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2e840cc200 con 0x7f2e70009ab0 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2e840cd2b0 con 0x7f2e840d0e60 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f2e897ce640 10 monclient(hunting): authenticate will time out at 1133.834704s 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 Processor -- start 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 -- start start 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 librados: setting wanted keys 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 1 librados: calling monclient init 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.254+0000 7f4ccbb96640 10 monclient: init 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4cc406d9d0 0x7f4cc4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc406d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4cc40c95a0 0x7f4cc41ee440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc40c95a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc41fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc4152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc406e9d0 con 0x7f4cc406d9d0 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc406e6f0 con 0x7f4cc4152ab0 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc4152e90 con 0x7f4cc40c95a0 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f4ccbb96640 10 monclient(hunting): authenticate will time out at 1133.835233s 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 10 monclient(hunting): handle_auth_done global_id 4453 payload 824 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 -- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f10181526a0 msgr2=0x7f10181474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f10181526a0 0x7f10181474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 -- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f10180cdb20 msgr2=0x7f1018144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f10180cdb20 0x7f1018144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7f101d487640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1018146150 con 0x7f1018152a80 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.255+0000 7fc176ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.256+0000 7fc1767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.256+0000 7fc1767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.256+0000 7fc1767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 10 monclient(hunting): handle_auth_done global_id 4456 payload 824 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 -- 10.20.193.51:0/4121244979 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc17815a420 msgr2=0x7fc17814f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc17815a420 0x7fc17814f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 -- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc17806d710 msgr2=0x7fc178073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc17806d710 0x7fc178073700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.257+0000 7fc1767fc640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1781f4300 con 0x7fc178153970 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f101d487640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1018152a80 0x7f1018147a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1014035000 tx=0x7f1014004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f100d7fa640 10 client.?.objecter ms_handle_connect 0x7f1018152a80 2026-02-22T21:37:38.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f100d7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905219920 0 0) 0x7f2fe4074040 con 0x7f2fe415d3b0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 10 monclient(hunting): my global_id is 4430 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.258+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe4073d60 con 0x7f2fe415d3b0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8046020 con 0x7fbc041526a0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: dump: 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8042420 con 0x7fbc041526a0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf803c040 con 0x7fbc041526a0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf1ffb640 10 monclient: dump: 2026-02-22T21:37:38.339 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbbf37fe640 4 set_mon_vals no callback set 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 5 monclient: authenticate success, global_id 4427 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 monclient: _renew_subs 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc04200e50 con 0x7fbc041526a0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 monclient: _renew_subs 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc04201410 con 0x7fbc041526a0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 1 librados: init done 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd53c0cc2f0 0x7fd53c1f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 client.4427.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd53c0cc2f0 0x7fd53c1f4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43664/0 (socket says 10.20.193.51:43664) 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 1 -- 10.20.193.51:0/1242726476 learned_addr learned my addr 10.20.193.51:0/1242726476 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 15 client.4427.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 20 client.4427.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbc0406a0b0 con 0x7fbc041526a0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 10 monclient(hunting): get_auth_request con 0x7fd53c0cc2f0 auth_method 0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.259+0000 7fd5411de640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8017070 con 0x7ff600150600 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: dump: 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e8005de0 con 0x7ff600150600 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8007290 con 0x7ff600150600 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 5 monclient: authenticate success, global_id 4425 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 monclient: _renew_subs 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5fd7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60014b470 con 0x7ff600150600 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff5df7fe640 10 monclient: dump: 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.341 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 monclient: _renew_subs 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60014ba30 con 0x7ff600150600 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 1 librados: init done 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7fd5411de640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7fd5411de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7fd5411de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0018020 con 0x7f73dc15aec0 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: dump: 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.342 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d0014480 con 0x7f73dc15aec0 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0014870 con 0x7f73dc15aec0 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 client.4425.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 15 client.4425.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 20 client.4425.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6000c73a0 con 0x7ff600150600 2026-02-22T21:37:38.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73b7fff640 10 monclient: dump: 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73d9ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 5 monclient: authenticate success, global_id 4437 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 10 monclient: _renew_subs 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc1f7ab0 con 0x7f73dc15aec0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 10 monclient: _renew_subs 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc1f8060 con 0x7f73dc15aec0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.260+0000 7f73e1f5c640 1 librados: init done 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 10 client.4437.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 15 client.4437.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 20 client.4437.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73dc0c8b00 con 0x7f73dc15aec0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 1 -- 10.20.193.51:0/1883859510 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283003c070 con 0x7f283c15da20 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: dump: 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.344 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 1 -- 10.20.193.51:0/1883859510 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2830037440 con 0x7f283c15da20 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 5 monclient: authenticate success, global_id 4434 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 1 -- 10.20.193.51:0/1883859510 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830037810 con 0x7f283c15da20 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 4 set_mon_vals no callback set 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f283a7fc640 10 monclient: dump: 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2736408242 0 0) 0x7f2fd4002920 con 0x7f2fe41526a0 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 1 -- 10.20.193.51:0/1638215907 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2803c070 con 0x7f7c4015c790 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: dump: 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.345 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 1 -- 10.20.193.51:0/1638215907 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c2802fad0 con 0x7f7c4015c790 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 1 -- 10.20.193.51:0/1638215907 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28037440 con 0x7f7c4015c790 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c3dffb640 10 monclient: dump: 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c45643640 5 monclient: authenticate success, global_id 4413 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c45643640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f7c45643640 4 set_mon_vals no callback set 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 10 monclient(hunting): my global_id is 4440 2026-02-22T21:37:38.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe4074040 con 0x7f2fe41526a0 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 10 monclient: shutdown 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 1 -- 10.20.193.51:0/1883859510 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c15da20 msgr2=0x7f283c0caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 1 --2- 10.20.193.51:0/1883859510 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c15da20 0x7f283c0caf90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f283c0751e0 tx=0x7f283002f2f0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 1 -- 10.20.193.51:0/1883859510 shutdown_connections 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.261+0000 7f2841b40640 1 --2- 10.20.193.51:0/1883859510 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c15da20 0x7f283c0caf90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 1 --2- 10.20.193.51:0/1883859510 >> v2:10.20.193.68:3300/0 conn(0x7f283c155ec0 0x7f283c15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 1 -- 10.20.193.51:0/1883859510 >> 10.20.193.51:0/1883859510 conn(0x7f283c0cc080 msgr2=0x7f283c0cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 1 -- 10.20.193.51:0/1883859510 shutdown_connections 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 1 -- 10.20.193.51:0/1883859510 wait complete. 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 10 monclient: shutdown 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 -- 10.20.193.51:0/1638215907 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c4015c790 msgr2=0x7f7c400c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 --2- 10.20.193.51:0/1638215907 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c4015c790 0x7f7c400c8d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c280098a0 tx=0x7f7c2802ee10 comp rx=0 tx=0).stop 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 -- 10.20.193.51:0/1638215907 shutdown_connections 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 --2- 10.20.193.51:0/1638215907 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c4015c790 0x7f7c400c8d60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 -- 10.20.193.51:0/1638215907 >> 10.20.193.51:0/1638215907 conn(0x7f7c4006d8c0 msgr2=0x7f7c4006dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 -- 10.20.193.51:0/1638215907 shutdown_connections 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 1 -- 10.20.193.51:0/1638215907 wait complete. 2026-02-22T21:37:38.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 0x7fd58c1fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014046020 con 0x7f1018152a80 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: dump: 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1014041450 con 0x7f1018152a80 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014037e60 con 0x7f1018152a80 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100d7fa640 10 monclient: dump: 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f100effd640 4 set_mon_vals no callback set 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 5 monclient: authenticate success, global_id 4453 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 10 monclient: _renew_subs 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1018146430 con 0x7f1018152a80 2026-02-22T21:37:38.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f2841b40640 10 monclient: monmap: 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 10 monclient: _renew_subs 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10181469f0 con 0x7f1018152a80 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f101ef11640 1 librados: init done 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152790 0x7fd53c0c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3676993272 0 0) 0x7f2fcc003620 con 0x7f2fe4152a80 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 >> v1:10.20.193.125:6789/0 conn(0x7f2fe415d3b0 legacy=0x7f2fe415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 >> v1:10.20.193.51:6789/0 conn(0x7f2fe41526a0 legacy=0x7f2fe40733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe4074320 con 0x7f2fe4152a80 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f2fd8002f00 con 0x7f2fe4152a80 2026-02-22T21:37:38.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7f7c45643640 10 monclient: monmap: 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 0x7fd58c1fbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60790/0 (socket says 10.20.193.51:60790) 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 1 -- 10.20.193.51:0/3368503904 learned_addr learned my addr 10.20.193.51:0/3368503904 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 10 monclient(hunting): get_auth_request con 0x7fd58c1526a0 auth_method 0 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.262+0000 7fd58b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: dump: 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 >> v1:10.20.193.68:6789/0 conn(0x7f2fe4152a80 legacy=0x7f2fe40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 librados: starting msgr at 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fcc003f90 0x7f2fcc004370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): picked mon.a con 0x7f2fcc003f90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fcc004a30 0x7f2fcc004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): picked mon.c con 0x7f2fcc004a30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 0x7f2fcc00bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): picked mon.b con 0x7f2fcc005520 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc00c440 con 0x7f2fcc003f90 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 librados: starting objecter 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc00c5f0 con 0x7f2fcc005520 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc00c7f0 con 0x7f2fcc004a30 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2fd3fff640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 0x7f2e840d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 Processor -- start 2026-02-22T21:37:38.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 librados: starting msgr at 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 -- start start 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 librados: starting objecter 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): get_auth_request con 0x7fd53c152790 auth_method 0 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 10 client.4453.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 15 client.4453.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 20 client.4453.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1018145b40 con 0x7f1018152a80 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 librados: setting wanted keys 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 librados: calling monclient init 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 Processor -- start 2026-02-22T21:37:38.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 10 monclient: init 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 0x7f2e840d0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60810/0 (socket says 10.20.193.51:60810) 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 1 -- 10.20.193.51:0/2582322711 learned_addr learned my addr 10.20.193.51:0/2582322711 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): get_auth_request con 0x7f2e841f3700 auth_method 0 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 -- start start 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 librados: setting wanted keys 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 1 librados: calling monclient init 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f7c45643640 10 monclient: init 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2e827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7fd5409dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c0ccba0 0x7f283c142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.263+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-a con 0x7f283c0ccba0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 0x7f283c1f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-b con 0x7f283c155ec0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f283c15da20 0x7f283c1f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): picked mon.noname-c con 0x7f283c15da20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c400c92a0 0x7f7c401f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f283c070010 con 0x7f283c0ccba0 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f283c06fd30 con 0x7f283c155ec0 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f283c0702f0 con 0x7f283c15da20 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f2841b40640 10 monclient(hunting): authenticate will time out at 1133.844216s 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-a con 0x7f7c400c92a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7c40155e60 0x7f7c40154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-c con 0x7f7c40155e60 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7c4015c790 0x7f7c40154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4015c790 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f7c400cccf0 con 0x7f7c400c92a0 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f7c400cca10 con 0x7f7c4015c790 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7c400ccfd0 con 0x7f7c40155e60 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.264+0000 7f7c45643640 10 monclient(hunting): authenticate will time out at 1133.844373s 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 -- 10.20.193.51:0/3368503904 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd58c152a80 msgr2=0x7fd58c1f9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd58c152a80 0x7fd58c1f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 -- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd58c0cdb20 msgr2=0x7fd58c1fb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd58c0cdb20 0x7fd58c1fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.265+0000 7fd58b7fe640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c1e9260 con 0x7fd58c1526a0 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 -- 10.20.193.51:0/2582322711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e840d0e60 msgr2=0x7f2e84148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2e840d0e60 0x7f2e84148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 -- 10.20.193.51:0/2582322711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70009ab0 msgr2=0x7f2e8415b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2e70009ab0 0x7f2e8415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7f2e827fc640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8414aad0 con 0x7f2e841f3700 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc98800a7e0 con 0x7fc99c0cdb20 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7fc977fff640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc96c078e50 0x7fc96c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7fc99affd640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc96c078e50 0x7fc96c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.267+0000 7fc99affd640 10 monclient: get_auth_request con 0x7fc96c078e50 auth_method 0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.268+0000 7fc99affd640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc96c078e50 0x7fc96c07b310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc98c039000 tx=0x7fc98c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.268+0000 7fc977fff640 10 client.4418.objecter ms_handle_connect 0x7fc96c078e50 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 -- 10.20.193.51:0/1242726476 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd53c0cc2f0 msgr2=0x7fd53c1f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd53c0cc2f0 0x7fd53c1f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 -- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd53c152b70 msgr2=0x7fd53c142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd53c152b70 0x7fd53c142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fd5409dd640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c144100 con 0x7fd53c152790 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fc98805d680 con 0x7fc99c0cdb20 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 10 client.4418.objecter ms_dispatch2 0x7fc99c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 3 client.4418.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 3 client.4418.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 10 client.4418.objecter checking command 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fc96c0813f0 0x7fc96c083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter _get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 15 client.4418.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter _get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 15 client.4418.objecter _session_command_op_assign 6 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 4 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 10 client.4418.objecter _send_command 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc96c083f00 con 0x7fc96c0813f0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.269+0000 7fc977fff640 20 client.4418.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.270+0000 7fc99bfff640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fc96c0813f0 0x7fc96c083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.270+0000 7fc99bfff640 10 monclient: get_auth_request con 0x7fc96c0813f0 auth_method 0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc98805da40 con 0x7fc99c0cdb20 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fc977fff640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 1 -- 10.20.193.51:0/2046266962 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67003c070 con 0x7fb67c152a80 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: dump: 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 1 -- 10.20.193.51:0/2046266962 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb67002fbe0 con 0x7fb67c152a80 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 1 -- 10.20.193.51:0/2046266962 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670037440 con 0x7fb67c152a80 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb66bfff640 10 monclient: dump: 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fd5409dd640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152790 0x7fd53c0c80a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd530036000 tx=0x7fd5300047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb6833ca640 5 monclient: authenticate success, global_id 4421 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb6833ca640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fb6833ca640 4 set_mon_vals no callback set 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fd529ffb640 10 client.?.objecter ms_handle_connect 0x7fd53c152790 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.271+0000 7fd529ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 10 monclient: shutdown 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 -- 10.20.193.51:0/2046266962 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 msgr2=0x7fb67c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 --2- 10.20.193.51:0/2046266962 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 0x7fb67c0cd5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb670009970 tx=0x7fb67002efb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fc977fff640 10 client.4418.objecter ms_handle_connect 0x7fc96c0813f0 2026-02-22T21:37:38.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fc99bfff640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fc96c0813f0 0x7fc96c083830 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 -- 10.20.193.51:0/2046266962 shutdown_connections 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 --2- 10.20.193.51:0/2046266962 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 0x7fb67c0cd5e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 -- 10.20.193.51:0/2046266962 >> 10.20.193.51:0/2046266962 conn(0x7fb67c0ce350 msgr2=0x7fb67c0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 -- 10.20.193.51:0/2046266962 shutdown_connections 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fb6833ca640 1 -- 10.20.193.51:0/2046266962 wait complete. 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.272+0000 7fc1767fc640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc178153970 0x7fc1780bff50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc16000e870 tx=0x7fc16000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fc14ffff640 10 client.?.objecter ms_handle_connect 0x7fc178153970 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fc14ffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient: monmap: 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.360 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 librados: starting msgr at 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 librados: starting objecter 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 Processor -- start 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 -- start start 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 librados: setting wanted keys 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 librados: calling monclient init 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient: init 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb67c0cdb20 0x7fb67c14cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c0cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 0x7fb67c1e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 0x7fb67c1ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c074600 con 0x7fb67c1526a0 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c074c50 con 0x7fb67c0cdb20 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c0748e0 con 0x7fb67c152a80 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.273+0000 7fb6833ca640 10 monclient(hunting): authenticate will time out at 1133.853969s 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 0x7fbe0c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 0x7fbe0c148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43696/0 (socket says 10.20.193.51:43696) 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 1 -- 10.20.193.51:0/2719535202 learned_addr learned my addr 10.20.193.51:0/2719535202 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 10 monclient(hunting): get_auth_request con 0x7fbe0c152a80 auth_method 0 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.275+0000 7fbe0b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.276+0000 7fbe0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.276+0000 7fbe0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.276+0000 7fbe0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7fd58b7fe640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 0x7fd58c1fbe10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00b500 tx=0x7fd57c00b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7fd588ff9640 10 client.?.objecter ms_handle_connect 0x7fd58c1526a0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7fd588ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7f2e827fc640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 0x7f2e840d0920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00d7a0 tx=0x7f2e6c00dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7f2e637fe640 10 client.?.objecter ms_handle_connect 0x7f2e841f3700 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.278+0000 7f2e637fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 -- 10.20.193.51:0/2719535202 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe0c15d3b0 msgr2=0x7fbe0c15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe0c15d3b0 0x7fbe0c15e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 -- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 msgr2=0x7fbe0c0ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe0c1526a0 0x7fbe0c0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7fbe0b7fe640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c0caf50 con 0x7fbe0c152a80 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.279+0000 7f6a68d52640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 0x7f6a641f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.281+0000 7f6a69553640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1014041840 con 0x7f1018152a80 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f100d7fa640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0fe8078e30 0x7f0fe807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f101cc86640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0fe8078e30 0x7f0fe807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f101cc86640 10 monclient: get_auth_request con 0x7f0fe8078e30 auth_method 0 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f101cc86640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0fe8078e30 0x7f0fe807b2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1008004660 tx=0x7f1008009210 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.282+0000 7f100d7fa640 10 client.4453.objecter ms_handle_connect 0x7f0fe8078e30 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f10140bd250 con 0x7f1018152a80 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 10 client.4453.objecter ms_dispatch2 0x7f1018000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 3 client.4453.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 3 client.4453.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 10 client.4453.objecter checking command 1 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7f0fe8081370 0x7f0fe80837b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter _get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 15 client.4453.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter _get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 15 client.4453.objecter _session_command_op_assign 14 1 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 0x7f6a641f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43692/0 (socket says 10.20.193.51:43692) 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 1 -- 10.20.193.51:0/1236403597 learned_addr learned my addr 10.20.193.51:0/1236403597 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 4 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 10 client.4453.objecter _send_command 1 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fe8083e80 con 0x7f0fe8081370 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 20 client.4453.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 10 monclient(hunting): get_auth_request con 0x7f6a640c9020 auth_method 0 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f6a68d52640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.284+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1014083d70 con 0x7f1018152a80 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f100d7fa640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f100ffff640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7f0fe8081370 0x7f0fe80837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a69553640 10 monclient(hunting): get_auth_request con 0x7f6a64156010 auth_method 0 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a69553640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a69553640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a69553640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f100ffff640 10 monclient: get_auth_request con 0x7f0fe8081370 auth_method 0 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f283affd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 0x7f283c1f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.285+0000 7f6a68d52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f6a69553640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f6a69553640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f6a69553640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7fbe0b7fe640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 0x7fbe0c148c70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00b5a0 tx=0x7fbdfc00ba70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7fbe08ff9640 10 client.?.objecter ms_handle_connect 0x7fbe0c152a80 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7fbe08ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 0x7f283c1f5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35422/0 (socket says 10.20.193.51:35422) 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 1 -- 10.20.193.51:0/1052472058 learned_addr learned my addr 10.20.193.51:0/1052472058 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): get_auth_request con 0x7f283c155ec0 auth_method 0 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc41fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 0x7f2fcc00bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc41fb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35402/0 (socket says 10.20.193.51:35402) 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 1 -- 10.20.193.51:0/3649964800 learned_addr learned my addr 10.20.193.51:0/3649964800 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 10 monclient(hunting): get_auth_request con 0x7f4cc4152ab0 auth_method 0 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f4cca10c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f2fe9efc640 10 monclient(hunting): get_auth_request con 0x7f2fcc005520 auth_method 0 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f2fe9efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f2fe9efc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f2fe9efc640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.286+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f4cca10c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f4cca10c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f4cca10c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f100d7fa640 10 client.4453.objecter ms_handle_connect 0x7f0fe8081370 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f100ffff640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7f0fe8081370 0x7f0fe80837b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.14 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f2fe9efc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f2fe9efc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f2fe9efc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 -- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 msgr2=0x7f6a640c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6a64156010 0x7f6a640c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 -- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a640caca0 msgr2=0x7f6a641517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6a640caca0 0x7f6a641517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.287+0000 7f6a68d52640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641f43b0 con 0x7f6a640c9020 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff5e8007430 con 0x7ff600150600 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 10 monclient(hunting): handle_auth_done global_id 4465 payload 824 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 -- 10.20.193.51:0/1052472058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f283c15da20 msgr2=0x7f283c1f5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f283c15da20 0x7f283c1f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 -- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c0ccba0 msgr2=0x7f283c142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f283c0ccba0 0x7f283c142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.288+0000 7f283affd640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c1512d0 con 0x7f283c155ec0 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.289+0000 7ff5df7fe640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff5cc078e50 0x7ff5cc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.289+0000 7ff5feffd640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff5cc078e50 0x7ff5cc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.289+0000 7ff5feffd640 10 monclient: get_auth_request con 0x7ff5cc078e50 auth_method 0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.289+0000 7ff5feffd640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff5cc078e50 0x7ff5cc07b310 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4039000 tx=0x7ff5f4031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.289+0000 7ff5df7fe640 10 client.4425.objecter ms_handle_connect 0x7ff5cc078e50 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 -- 10.20.193.51:0/3649964800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4cc40c95a0 msgr2=0x7f4cc41ee440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4cc40c95a0 0x7f4cc41ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 -- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4cc406d9d0 msgr2=0x7f4cc4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4cc406d9d0 0x7f4cc4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.290+0000 7f4cca10c640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc4153690 con 0x7f4cc4152ab0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.291+0000 7f6a68d52640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 0x7f6a641f1630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009a70 tx=0x7f6a540047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.291+0000 7f6a617fa640 10 client.?.objecter ms_handle_connect 0x7f6a640c9020 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.291+0000 7f6a617fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 10 monclient(hunting): handle_auth_done global_id 4450 payload 293 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 -- 10.20.193.51:0/1907496938 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fcc004a30 msgr2=0x7f2fcc004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fcc004a30 0x7f2fcc004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 -- 10.20.193.51:0/1907496938 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fcc003f90 msgr2=0x7f2fcc004370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fcc003f90 0x7f2fcc004370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 monclient: found mon.b 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 -- 10.20.193.51:0/1907496938 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc003620 con 0x7f2fcc005520 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2feb986640 5 monclient: authenticate success, global_id 4450 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2feb986640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7ff5e8007750 con 0x7ff600150600 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 10 client.4425.objecter ms_dispatch2 0x7ff600000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 3 client.4425.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 3 client.4425.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 10 client.4425.objecter checking command 1 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7ff5cc0813f0 0x7ff5cc083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter _get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 15 client.4425.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter _get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 15 client.4425.objecter _session_command_op_assign 3 1 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 4 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 10 client.4425.objecter _send_command 1 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5e8020ec0 con 0x7ff5cc0813f0 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 20 client.4425.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5e805daa0 con 0x7ff600150600 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5df7fe640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7f2fe9efc640 1 -- 10.20.193.51:0/1907496938 reap_dead start 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.292+0000 7ff5ff7fe640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7ff5cc0813f0 0x7ff5cc083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.293+0000 7ff5ff7fe640 10 monclient: get_auth_request con 0x7ff5cc0813f0 auth_method 0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f283affd640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 0x7f283c1f5640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f282c00b570 tx=0x7f282c00ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f2817fff640 10 client.?.objecter ms_handle_connect 0x7f283c155ec0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f2817fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f7c3e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7c40155e60 0x7f7c40154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f4cca10c640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc41fb2b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000ba10 tx=0x7f4cc000bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f4cb27fc640 10 client.?.objecter ms_handle_connect 0x7f4cc4152ab0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f4cb27fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 0x7f2fcc00bb90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c2f0 tx=0x7f2fe002f630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=539de32f76e65699 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== osd.6 v2:10.20.193.125:6824/1184344866 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fc96c083f00 con 0x7fc96c0813f0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7fc977fff640 10 client.4418.objecter ms_dispatch2 0x7fc99c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7fc977fff640 10 client.4418.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7fc977fff640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 4 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.294+0000 7fc977fff640 15 client.4418.objecter _session_command_op_remove 6 1 2026-02-22T21:37:38.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7c40155e60 0x7f7c40154000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43726/0 (socket says 10.20.193.51:43726) 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 1 -- 10.20.193.51:0/1932056164 learned_addr learned my addr 10.20.193.51:0/1932056164 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c40155e60 auth_method 0 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7f7c3f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7c4015c790 0x7f7c40154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7ff5df7fe640 10 client.4425.objecter ms_handle_connect 0x7ff5cc0813f0 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7ff5ff7fe640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7ff5cc0813f0 0x7ff5cc083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc012070 con 0x7fbe0c152a80 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: dump: 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.371 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdfc0044a0 con 0x7fbe0c152a80 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc0029c0 con 0x7fbe0c152a80 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe08ff9640 10 monclient: dump: 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe1295c640 5 monclient: authenticate success, global_id 4436 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe1295c640 10 monclient: _renew_subs 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe0a7fc640 4 set_mon_vals no callback set 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe1295c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.295+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c1f3b00 con 0x7fbe0c152a80 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 monclient: _renew_subs 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c1f40c0 con 0x7fbe0c152a80 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 1 librados: init done 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7f7c3e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 client.4436.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 15 client.4436.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 20 client.4436.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.296+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe0c15d820 con 0x7fbe0c152a80 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.298+0000 7f7c3f7fe640 10 monclient(hunting): get_auth_request con 0x7f7c4015c790 auth_method 0 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.298+0000 7f7c3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.298+0000 7f7c3f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.298+0000 7f7c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.299+0000 7f7c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.299+0000 7f7c3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.299+0000 7f7c3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 10 monclient(hunting): handle_auth_done global_id 4471 payload 824 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 -- 10.20.193.51:0/1932056164 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7c40155e60 msgr2=0x7f7c40154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7c40155e60 0x7f7c40154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 -- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c400c92a0 msgr2=0x7f7c401f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7c400c92a0 0x7f7c401f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.300+0000 7f7c3f7fe640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c40151e80 con 0x7f7c4015c790 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c013020 con 0x7f283c155ec0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: dump: 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f282c0044a0 con 0x7f283c155ec0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 5 monclient: authenticate success, global_id 4465 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 monclient: _renew_subs 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f283c1515b0 con 0x7f283c155ec0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c0029c0 con 0x7f283c155ec0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2817fff640 10 monclient: dump: 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.374 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2839ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 monclient: _renew_subs 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f283c1f4e40 con 0x7f283c155ec0 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 1 librados: init done 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 client.4465.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 15 client.4465.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 20 client.4465.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.301+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f283c0705d0 con 0x7f283c155ec0 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 10 client.4418.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 15 client.4418.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 20 client.4418.objecter _get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 20 client.4418.objecter _get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 15 client.4418.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 20 client.4418.objecter get_session s=0x7fc96c080d20 osd=6 3 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 15 client.4418.objecter _session_command_op_assign 6 2 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 4 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 10 client.4418.objecter _send_command 2 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160017070 con 0x7fc178153970 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 --> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc99c150840 con 0x7fc96c0813f0 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: dump: 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1600098c0 con 0x7fc178153970 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16000f040 con 0x7fc178153970 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc14ffff640 10 monclient: dump: 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 5 monclient: authenticate success, global_id 4456 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 10 monclient: _renew_subs 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc1757fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1781f4020 con 0x7fc178153970 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 10 monclient: _renew_subs 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc178072c20 con 0x7fc178153970 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.302+0000 7fc17d944640 1 librados: init done 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 10 client.4456.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 15 client.4456.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 20 client.4456.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1780c5b00 con 0x7fc178153970 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00038a0 con 0x7f2fcc005520 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: dump: 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe0037630 con 0x7f2fcc005520 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 1 -- 10.20.193.51:0/1907496938 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0037a20 con 0x7f2fcc005520 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2fd3fff640 10 monclient: dump: 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.377 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 4 set_mon_vals no callback set 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f73d0014a10 con 0x7f73dc15aec0 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fb68113f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb67c0cdb20 0x7fb67c14cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f73b7fff640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f73ac078e50 0x7f73ac07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f7c3f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7c4015c790 0x7f7c40154540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000caa0 tx=0x7f7c3000cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f7c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c4015c790 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f7c1f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 10 monclient: shutdown 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 -- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 msgr2=0x7f2fcc00bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 0x7f2fcc00bb90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c2f0 tx=0x7f2fe002f630 comp rx=0 tx=0).stop 2026-02-22T21:37:38.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f73db7fe640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f73ac078e50 0x7f73ac07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f73db7fe640 10 monclient: get_auth_request con 0x7f73ac078e50 auth_method 0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fbbf803f030 con 0x7fbc041526a0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 -- 10.20.193.51:0/1907496938 shutdown_connections 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 --2- 10.20.193.51:0/1907496938 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fcc005520 0x7f2fcc00bb90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 -- 10.20.193.51:0/1907496938 >> 10.20.193.51:0/1907496938 conn(0x7f2fe40ce350 msgr2=0x7f2fe40ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.303+0000 7f2feb986640 1 -- 10.20.193.51:0/1907496938 shutdown_connections 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 -- 10.20.193.51:0/1907496938 wait complete. 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f73b7fff640 10 client.4437.objecter ms_handle_connect 0x7f73ac078e50 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f73db7fe640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f73ac078e50 0x7f73ac07b310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73dc1566b0 tx=0x7f73c8039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7fbbf1ffb640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbd4078e50 0x7fbbd407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7fbc0914a640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbd4078e50 0x7fbbd407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7fbc0914a640 10 monclient: get_auth_request con 0x7fbbd4078e50 auth_method 0 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7fbbf1ffb640 10 client.4427.objecter ms_handle_connect 0x7fbbd4078e50 2026-02-22T21:37:38.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7fbc0914a640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbd4078e50 0x7fbbd407b310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009750 tx=0x7fbbf4006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 10 monclient: monmap: 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 librados: starting msgr at 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 librados: starting objecter 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 Processor -- start 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 -- start start 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 librados: setting wanted keys 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 1 librados: calling monclient init 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.304+0000 7f2feb986640 10 monclient: init 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 0x7f2fe41496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-a con 0x7f2fe4143c90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fe414d5e0 0x7f2fe414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-c con 0x7f2fe414d5e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fe41f3ad0 0x7f2fe41f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): picked mon.noname-b con 0x7f2fe41f3ad0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe4074c50 con 0x7f2fe4143c90 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe40cca00 con 0x7f2fe41f3ad0 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe40748e0 con 0x7f2fe414d5e0 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7f2feb986640 10 monclient(hunting): authenticate will time out at 1133.885418s 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc977fff640 1 -- 10.20.193.51:0/225000125 <== osd.6 v2:10.20.193.125:6824/1184344866 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc99c150840 con 0x7fc96c0813f0 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc977fff640 10 client.4418.objecter ms_dispatch2 0x7fc99c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc977fff640 10 client.4418.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc977fff640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 4 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc977fff640 15 client.4418.objecter _session_command_op_remove 6 2 2026-02-22T21:37:38.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 10 librados: watch_flush enter 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 10 librados: watch_flush exit 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 10 client.4418.objecter close_session for osd.6 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fc96c0813f0 msgr2=0x7fc96c083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fc96c0813f0 0x7fc96c083830 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 20 client.4418.objecter put_session s=0x7fc96c080d20 osd=6 1 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 20 client.4418.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 10 client.4418.objecter successfully canceled tick 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc96c078e50 msgr2=0x7fc96c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc96c078e50 0x7fc96c07b310 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc98c039000 tx=0x7fc98c033000 comp rx=0 tx=0).stop 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 10 monclient: shutdown 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 msgr2=0x7fc99c1f7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 --2- 10.20.193.51:0/225000125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc99c0cdb20 0x7fc99c1f7850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc98800c700 tx=0x7fc98800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc99bfff640 1 -- 10.20.193.51:0/225000125 reap_dead start 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 shutdown_connections 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 >> 10.20.193.51:0/225000125 conn(0x7fc99c0ce350 msgr2=0x7fc99c151640 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 shutdown_connections 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.305+0000 7fc9a1d85640 1 -- 10.20.193.51:0/225000125 wait complete. 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fc9a1d85640 1 librados: shutdown 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fbbf80832e0 con 0x7fbc041526a0 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 10 client.4427.objecter ms_dispatch2 0x7fbc04000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 3 client.4427.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 3 client.4427.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 10 client.4427.objecter checking command 1 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7fbbd40813f0 0x7fbbd4083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter _get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 15 client.4427.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter _get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 15 client.4427.objecter _session_command_op_assign 5 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 4 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 10 client.4427.objecter _send_command 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbf8036ec0 con 0x7fbbd40813f0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 20 client.4427.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f73d005f260 con 0x7f73dc15aec0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 10 client.4437.objecter ms_dispatch2 0x7f73dc000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 3 client.4437.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 3 client.4437.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 0x7fb67c1e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 10 client.4437.objecter checking command 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f73ac081390 0x7f73ac0837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter _get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 15 client.4437.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter _get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 15 client.4437.objecter _session_command_op_assign 10 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 4 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 10 client.4437.objecter _send_command 1 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73d0023e50 con 0x7f73ac081390 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 20 client.4437.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f73d005f620 con 0x7f73dc15aec0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7f73b7fff640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbbf80836a0 con 0x7fbc041526a0 2026-02-22T21:37:38.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbbf1ffb640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 0x7fb67c1e9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60850/0 (socket says 10.20.193.51:60850) 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 1 -- 10.20.193.51:0/3929518326 learned_addr learned my addr 10.20.193.51:0/3929518326 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 10 monclient(hunting): get_auth_request con 0x7fb67c1526a0 auth_method 0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fb68093e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.306+0000 7fbc0994b640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7fbbd40813f0 0x7fbbd4083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7f73daffd640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f73ac081390 0x7f73ac0837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7fbc0994b640 10 monclient: get_auth_request con 0x7fbbd40813f0 auth_method 0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7f73daffd640 10 monclient: get_auth_request con 0x7f73ac081390 auth_method 0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7fb68093e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.307+0000 7f2fe96fb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 0x7f2fe41496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.308+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== osd.14 v2:10.20.193.125:6816/498199928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f0fe8083e80 con 0x7f0fe8081370 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.308+0000 7f100d7fa640 10 client.4453.objecter ms_dispatch2 0x7f1018000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.308+0000 7f100d7fa640 10 client.4453.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.308+0000 7f100d7fa640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 4 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.308+0000 7f100d7fa640 15 client.4453.objecter _session_command_op_remove 14 1 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7f73daffd640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f73ac081390 0x7f73ac0837d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7f73b7fff640 10 client.4437.objecter ms_handle_connect 0x7f73ac081390 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fbc0994b640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7fbbd40813f0 0x7fbbd4083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fbbf1ffb640 10 client.4427.objecter ms_handle_connect 0x7fbbd40813f0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 -- 10.20.193.51:0/3929518326 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 msgr2=0x7fb67c1ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb67c152a80 0x7fb67c1ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 -- 10.20.193.51:0/3929518326 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb67c0cdb20 msgr2=0x7fb67c14cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb67c0cdb20 0x7fb67c14cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.309+0000 7fb68093e640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c1553e0 con 0x7fb67c1526a0 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 0x7f2fe41496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60864/0 (socket says 10.20.193.51:60864) 2026-02-22T21:37:38.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 1 -- 10.20.193.51:0/192091311 learned_addr learned my addr 10.20.193.51:0/192091311 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 10 monclient(hunting): get_auth_request con 0x7f2fe4143c90 auth_method 0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7f2fe96fb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c004480 con 0x7fd58c1526a0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: dump: 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c014070 con 0x7fd58c1526a0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c002920 con 0x7fd58c1526a0 2026-02-22T21:37:38.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 5 monclient: authenticate success, global_id 4449 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd58a7fc640 4 set_mon_vals no callback set 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd588ff9640 10 monclient: dump: 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 10 monclient: _renew_subs 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c1e8400 con 0x7fd58c1526a0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 10 monclient: _renew_subs 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c1e89c0 con 0x7fd58c1526a0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.310+0000 7fd59231b640 1 librados: init done 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 10 client.4449.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 15 client.4449.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 20 client.4449.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd58c0cd5e0 con 0x7fd58c1526a0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7f2fe96fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7f2fe96fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7f2fe96fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fb68093e640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 0x7fb67c1e9af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb670009aa0 tx=0x7fb6700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fb669ffb640 10 client.?.objecter ms_handle_connect 0x7fb67c1526a0 2026-02-22T21:37:38.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.311+0000 7fb669ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.388 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 -- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fe414d5e0 msgr2=0x7f2fe414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2fe414d5e0 0x7f2fe414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 -- 10.20.193.51:0/192091311 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fe41f3ad0 msgr2=0x7f2fe41f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2fe41f3ad0 0x7f2fe41f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.313+0000 7f2fe96fb640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe414dfa0 con 0x7f2fe4143c90 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 10 client.4453.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 15 client.4453.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 20 client.4453.objecter _get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 20 client.4453.objecter _get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 15 client.4453.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 20 client.4453.objecter get_session s=0x7f0fe8080ca0 osd=14 3 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 15 client.4453.objecter _session_command_op_assign 14 2 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 4 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 10 client.4453.objecter _send_command 2 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.314+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 --> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f10180d0930 con 0x7f0fe8081370 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f282c00f460 con 0x7f283c155ec0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f2817fff640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f280c078e50 0x7f280c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f283b7fe640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f280c078e50 0x7f280c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f283b7fe640 10 monclient: get_auth_request con 0x7f280c078e50 auth_method 0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f2817fff640 10 client.4465.objecter ms_handle_connect 0x7f280c078e50 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.315+0000 7f283b7fe640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f280c078e50 0x7f280c07b310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f28280096b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f282c00c250 con 0x7f283c155ec0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 10 client.4465.objecter ms_dispatch2 0x7f283c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 3 client.4465.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 3 client.4465.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 10 client.4465.objecter checking command 1 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f280c081390 0x7f280c0837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter _get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 15 client.4465.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter _get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 15 client.4465.objecter _session_command_op_assign 12 1 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 4 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 10 client.4465.objecter _send_command 1 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f282c01be50 con 0x7f280c081390 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.317+0000 7f2817fff640 20 client.4465.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== osd.3 v2:10.20.193.140:6824/3567184175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7ff5e8020ec0 con 0x7ff5cc0813f0 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7ff5df7fe640 10 client.4425.objecter ms_dispatch2 0x7ff600000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7ff5df7fe640 10 client.4425.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7ff5df7fe640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 4 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7ff5df7fe640 15 client.4425.objecter _session_command_op_remove 3 1 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f282c05e4d0 con 0x7f283c155ec0 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7f2817fff640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7f283bfff640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f280c081390 0x7f280c0837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.318+0000 7f283bfff640 10 monclient: get_auth_request con 0x7f280c081390 auth_method 0 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530046020 con 0x7fd53c152790 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: dump: 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd530042620 con 0x7fd53c152790 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53003c040 con 0x7fd53c152790 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd529ffb640 10 monclient: dump: 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd52b7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 5 monclient: authenticate success, global_id 4459 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 monclient: _renew_subs 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53c0d0360 con 0x7fd53c152790 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 monclient: _renew_subs 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd53c1443c0 con 0x7fd53c152790 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 1 librados: init done 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.319+0000 7fd543469640 10 client.4459.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7fd543469640 15 client.4459.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7fd543469640 20 client.4459.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7fd543469640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7fd543469640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd53c0cbfc0 con 0x7fd53c152790 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f100d7fa640 1 -- 10.20.193.51:0/225282276 <== osd.14 v2:10.20.193.125:6816/498199928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f10180d0930 con 0x7f0fe8081370 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f100d7fa640 10 client.4453.objecter ms_dispatch2 0x7f1018000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f100d7fa640 10 client.4453.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f100d7fa640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 4 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f100d7fa640 15 client.4453.objecter _session_command_op_remove 14 2 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 10 librados: watch_flush enter 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 10 librados: watch_flush exit 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 10 client.4453.objecter close_session for osd.14 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7f0fe8081370 msgr2=0x7f0fe80837b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7f0fe8081370 0x7f0fe80837b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 20 client.4453.objecter put_session s=0x7f0fe8080ca0 osd=14 1 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 20 client.4453.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 10 client.4453.objecter successfully canceled tick 2026-02-22T21:37:38.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0fe8078e30 msgr2=0x7f0fe807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0fe8078e30 0x7f0fe807b2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1008004660 tx=0x7f1008009210 comp rx=0 tx=0).stop 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 10 monclient: shutdown 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1018152a80 msgr2=0x7f1018147a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 --2- 10.20.193.51:0/225282276 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1018152a80 0x7f1018147a10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1014035000 tx=0x7f1014004b40 comp rx=0 tx=0).stop 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f2817fff640 10 client.4465.objecter ms_handle_connect 0x7f280c081390 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f283bfff640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f280c081390 0x7f280c0837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.12 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101d487640 1 -- 10.20.193.51:0/225282276 reap_dead start 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 shutdown_connections 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 >> 10.20.193.51:0/225282276 conn(0x7f10180ce350 msgr2=0x7f10180ccc00 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 shutdown_connections 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 -- 10.20.193.51:0/225282276 wait complete. 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.320+0000 7f101ef11640 1 librados: shutdown 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fbdfc00d460 con 0x7fbe0c152a80 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 10 client.4425.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 15 client.4425.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 20 client.4425.objecter _get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 20 client.4425.objecter _get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 15 client.4425.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 20 client.4425.objecter get_session s=0x7ff5cc080d20 osd=3 3 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 15 client.4425.objecter _session_command_op_assign 3 2 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 4 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 10 client.4425.objecter _send_command 2 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.324+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 --> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff60006ced0 con 0x7ff5cc0813f0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.325+0000 7fbe08ff9640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbddc078e50 0x7fbddc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.325+0000 7fbe0bfff640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbddc078e50 0x7fbddc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.325+0000 7fbe0bfff640 10 monclient: get_auth_request con 0x7fbddc078e50 auth_method 0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.325+0000 7fbe0bfff640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbddc078e50 0x7fbddc07b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c15f410 tx=0x7fbdf8031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.325+0000 7fbe08ff9640 10 client.4436.objecter ms_handle_connect 0x7fbddc078e50 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd57c019460 con 0x7fd58c1526a0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fbdfc05db20 con 0x7fbe0c152a80 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 10 client.4436.objecter ms_dispatch2 0x7fbe0c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 3 client.4436.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 3 client.4436.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 10 client.4436.objecter checking command 1 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7fbddc081390 0x7fbddc0837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7ff5df7fe640 1 -- 10.20.193.51:0/3356402320 <== osd.3 v2:10.20.193.140:6824/3567184175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff60006ced0 con 0x7ff5cc0813f0 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter _get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 15 client.4436.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter _get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 15 client.4436.objecter _session_command_op_assign 0 1 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 4 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 10 client.4436.objecter _send_command 1 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7ff5df7fe640 10 client.4425.objecter ms_dispatch2 0x7ff600000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7ff5df7fe640 10 client.4425.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7ff5df7fe640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 4 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7ff5df7fe640 15 client.4425.objecter _session_command_op_remove 3 2 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdfc018e50 con 0x7fbddc081390 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fbe08ff9640 20 client.4436.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.327+0000 7fd588ff9640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd55c078e50 0x7fd55c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbdfc05dee0 con 0x7fbe0c152a80 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fbe08ff9640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 10 librados: watch_flush enter 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 10 librados: watch_flush exit 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 10 client.4425.objecter close_session for osd.3 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7ff5cc0813f0 msgr2=0x7ff5cc083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7ff5cc0813f0 0x7ff5cc083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 20 client.4425.objecter put_session s=0x7ff5cc080d20 osd=3 1 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 20 client.4425.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 10 client.4425.objecter successfully canceled tick 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fd58bfff640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd55c078e50 0x7fd55c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fd58bfff640 10 monclient: get_auth_request con 0x7fd55c078e50 auth_method 0 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff5cc078e50 msgr2=0x7ff5cc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff5cc078e50 0x7ff5cc07b310 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4039000 tx=0x7ff5f4031000 comp rx=0 tx=0).stop 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 10 monclient: shutdown 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 msgr2=0x7ff60006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 --2- 10.20.193.51:0/3356402320 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff600150600 0x7ff60006d170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c4d0 tx=0x7ff5e800c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff5ff7fe640 1 -- 10.20.193.51:0/3356402320 reap_dead start 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 shutdown_connections 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 >> 10.20.193.51:0/3356402320 conn(0x7ff6000cd7b0 msgr2=0x7ff6000cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 shutdown_connections 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 -- 10.20.193.51:0/3356402320 wait complete. 2026-02-22T21:37:38.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7ff605355640 1 librados: shutdown 2026-02-22T21:37:38.397 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:38.397 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2026-02-22T21:37:38.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fbe10ed2640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7fbddc081390 0x7fbddc0837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fd58bfff640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd55c078e50 0x7fd55c07b310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd578009750 tx=0x7fd578006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fd588ff9640 10 client.4449.objecter ms_handle_connect 0x7fd55c078e50 2026-02-22T21:37:38.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7fbe10ed2640 10 monclient: get_auth_request con 0x7fbddc081390 auth_method 0 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54047070 con 0x7f6a640c9020 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: dump: 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a54042630 con 0x7f6a640c9020 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 5 monclient: authenticate success, global_id 4439 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 10 monclient: _renew_subs 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6414eff0 con 0x7f6a640c9020 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5403c040 con 0x7f6a640c9020 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a62ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a617fa640 10 monclient: dump: 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.398 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 10 monclient: _renew_subs 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6414f5b0 con 0x7f6a640c9020 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.328+0000 7f6a69d54640 1 librados: init done 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 10 client.4439.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 15 client.4439.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 20 client.4439.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.329+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a641563f0 con 0x7f6a640c9020 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.330+0000 7fbe10ed2640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7fbddc081390 0x7fbddc0837d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.330+0000 7fbe08ff9640 10 client.4436.objecter ms_handle_connect 0x7fbddc081390 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== osd.5 v2:10.20.193.68:6816/3736303751 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fbbf8036ec0 con 0x7fbbd40813f0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fbbf1ffb640 10 client.4427.objecter ms_dispatch2 0x7fbc04000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fbbf1ffb640 10 client.4427.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fbbf1ffb640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 4 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fbbf1ffb640 15 client.4427.objecter _session_command_op_remove 5 1 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fd57c05dd00 con 0x7fd58c1526a0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 10 client.4449.objecter ms_dispatch2 0x7fd58c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 3 client.4449.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 3 client.4449.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 10 client.4449.objecter checking command 1 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fd55c0813f0 0x7fd55c083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter _get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 15 client.4449.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter _get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 15 client.4449.objecter _session_command_op_assign 13 1 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 4 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 10 client.4449.objecter _send_command 1 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd55c083f00 con 0x7fd55c0813f0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 20 client.4449.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd57c05e0c0 con 0x7fd58c1526a0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7fd588ff9640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7f2fe96fb640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 0x7f2fe41496d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c760 tx=0x7f2fd400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7f2fd1ffb640 10 client.?.objecter ms_handle_connect 0x7f2fe4143c90 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.332+0000 7f2fd1ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7fd590891640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fd55c0813f0 0x7fd55c083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== osd.10 v2:10.20.193.125:6800/1951583296 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f73d0023e50 con 0x7f73ac081390 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7f73b7fff640 10 client.4437.objecter ms_dispatch2 0x7f73dc000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7f73b7fff640 10 client.4437.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7f73b7fff640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 4 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7f73b7fff640 15 client.4437.objecter _session_command_op_remove 10 1 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.333+0000 7fd590891640 10 monclient: get_auth_request con 0x7fd55c0813f0 auth_method 0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7fd588ff9640 10 client.4449.objecter ms_handle_connect 0x7fd55c0813f0 2026-02-22T21:37:38.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7fd590891640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fd55c0813f0 0x7fd55c083830 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.13 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c005270 con 0x7f2e841f3700 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: dump: 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6c017070 con 0x7f2e841f3700 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c003710 con 0x7f2e841f3700 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 5 monclient: authenticate success, global_id 4452 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 10 monclient: _renew_subs 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e817fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e841f3c30 con 0x7f2e841f3700 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e637fe640 10 monclient: dump: 2026-02-22T21:37:38.401 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 10 monclient: _renew_subs 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e841f41f0 con 0x7f2e841f3700 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.335+0000 7f2e897ce640 1 librados: init done 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.402 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 10 client.4452.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 15 client.4452.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 20 client.4452.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e8415cbd0 con 0x7f2e841f3700 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.336+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc160012050 con 0x7fc178153970 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40052d0 con 0x7f2fe4143c90 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: dump: 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd4005490 con 0x7f2fe4143c90 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40038e0 con 0x7f2fe4143c90 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 5 monclient: authenticate success, global_id 4458 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 monclient: _renew_subs 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe41f4490 con 0x7f2fe4143c90 2026-02-22T21:37:38.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd37fe640 4 set_mon_vals no callback set 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2fd1ffb640 10 monclient: dump: 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 monclient: _renew_subs 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe414c820 con 0x7f2fe4143c90 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 1 librados: init done 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670044070 con 0x7fb67c1526a0 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: dump: 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb670040610 con 0x7fb67c1526a0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67003c040 con 0x7fb67c1526a0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 5 monclient: authenticate success, global_id 4455 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb66b7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 10 monclient: _renew_subs 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fc14ffff640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc148078e50 0x7fc14807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c1556c0 con 0x7fb67c1526a0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb669ffb640 10 monclient: dump: 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 10 monclient: _renew_subs 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c155c80 con 0x7fb67c1526a0 2026-02-22T21:37:38.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fb6833ca640 1 librados: init done 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fc176ffd640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc148078e50 0x7fc14807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fc176ffd640 10 monclient: get_auth_request con 0x7fc148078e50 auth_method 0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 client.4458.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 15 client.4458.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 20 client.4458.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2fe415d820 con 0x7f2fe4143c90 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fc176ffd640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc148078e50 0x7fc14807b310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc16c02eef0 tx=0x7fc16c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.337+0000 7fc14ffff640 10 client.4456.objecter ms_handle_connect 0x7fc148078e50 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 10 client.4455.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 15 client.4455.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 20 client.4455.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.338+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb67c0cd5e0 con 0x7fb67c1526a0 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 10 client.4427.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 15 client.4427.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 20 client.4427.objecter _get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 20 client.4427.objecter _get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 15 client.4427.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fc16005d930 con 0x7fc178153970 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 10 client.4456.objecter ms_dispatch2 0x7fc178000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 3 client.4456.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 3 client.4456.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 20 client.4427.objecter get_session s=0x7fbbd4080d20 osd=5 3 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 15 client.4427.objecter _session_command_op_assign 5 2 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 4 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 10 client.4427.objecter _send_command 2 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 --> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbc04150c50 con 0x7fbbd40813f0 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 10 client.4456.objecter checking command 1 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7fc148081390 0x7fc1480837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter _get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 15 client.4456.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter _get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 15 client.4456.objecter _session_command_op_assign 8 1 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 4 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 10 client.4456.objecter _send_command 1 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc16001fe70 con 0x7fc148081390 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 20 client.4456.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc16005dcf0 con 0x7fc178153970 2026-02-22T21:37:38.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.339+0000 7fc14ffff640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 10 client.4437.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 15 client.4437.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 20 client.4437.objecter _get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 20 client.4437.objecter _get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 15 client.4437.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 20 client.4437.objecter get_session s=0x7f73ac080cc0 osd=10 3 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 15 client.4437.objecter _session_command_op_assign 10 2 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 4 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 10 client.4437.objecter _send_command 2 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.340+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 --> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f73dc0cacc0 con 0x7f73ac081390 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.341+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6a5403f3c0 con 0x7f6a640c9020 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.341+0000 7f6a617fa640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a3c078e50 0x7f6a3c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.341+0000 7f6a63fff640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a3c078e50 0x7f6a3c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.341+0000 7f6a63fff640 10 monclient: get_auth_request con 0x7f6a3c078e50 auth_method 0 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7f6a617fa640 10 client.4439.objecter ms_handle_connect 0x7f6a3c078e50 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7f6a63fff640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a3c078e50 0x7f6a3c07b310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006b90 tx=0x7f6a50008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbbf1ffb640 1 -- 10.20.193.51:0/4034623388 <== osd.5 v2:10.20.193.68:6816/3736303751 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbc04150c50 con 0x7fbbd40813f0 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbbf1ffb640 10 client.4427.objecter ms_dispatch2 0x7fbc04000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbbf1ffb640 10 client.4427.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbbf1ffb640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 4 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbbf1ffb640 15 client.4427.objecter _session_command_op_remove 5 2 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.342+0000 7fbc0b3d5640 10 librados: watch_flush enter 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 10 librados: watch_flush exit 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 10 client.4427.objecter close_session for osd.5 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7fbbd40813f0 msgr2=0x7fbbd4083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7fbbd40813f0 0x7fbbd4083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 20 client.4427.objecter put_session s=0x7fbbd4080d20 osd=5 1 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 20 client.4427.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 10 client.4427.objecter successfully canceled tick 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbd4078e50 msgr2=0x7fbbd407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbd4078e50 0x7fbbd407b310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009750 tx=0x7fbbf4006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 10 monclient: shutdown 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 msgr2=0x7fbc0414b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 --2- 10.20.193.51:0/4034623388 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc041526a0 0x7fbc0414b1c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8002410 tx=0x7fbbf80047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0994b640 1 -- 10.20.193.51:0/4034623388 reap_dead start 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 shutdown_connections 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 >> 10.20.193.51:0/4034623388 conn(0x7fbc040ce350 msgr2=0x7fbc04151500 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 shutdown_connections 2026-02-22T21:37:38.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 -- 10.20.193.51:0/4034623388 wait complete. 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== osd.12 v2:10.20.193.51:6818/1676524147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f282c01be50 con 0x7f280c081390 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f2817fff640 10 client.4465.objecter ms_dispatch2 0x7f283c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f2817fff640 10 client.4465.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f2817fff640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 4 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f2817fff640 15 client.4465.objecter _session_command_op_remove 12 1 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7fbc0b3d5640 1 librados: shutdown 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f6a5402fa40 con 0x7f6a640c9020 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 10 client.4439.objecter ms_dispatch2 0x7f6a64000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 3 client.4439.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 3 client.4439.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 10 client.4439.objecter checking command 1 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f6a3c0813f0 0x7f6a3c083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter _get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 15 client.4439.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter _get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 15 client.4439.objecter _session_command_op_assign 9 1 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 4 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 10 client.4439.objecter _send_command 1 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a54043e50 con 0x7f6a3c0813f0 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.343+0000 7f6a617fa640 20 client.4439.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f6a69553640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f6a3c0813f0 0x7f6a3c083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f6a69553640 10 monclient: get_auth_request con 0x7f6a3c0813f0 auth_method 0 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73b7fff640 1 -- 10.20.193.51:0/984994084 <== osd.10 v2:10.20.193.125:6800/1951583296 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f73dc0cacc0 con 0x7f73ac081390 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73b7fff640 10 client.4437.objecter ms_dispatch2 0x7f73dc000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73b7fff640 10 client.4437.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73b7fff640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 4 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73b7fff640 15 client.4437.objecter _session_command_op_remove 10 2 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 10 librados: watch_flush enter 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 10 librados: watch_flush exit 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 10 client.4437.objecter close_session for osd.10 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f73ac081390 msgr2=0x7f73ac0837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f73ac081390 0x7f73ac0837d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 20 client.4437.objecter put_session s=0x7f73ac080cc0 osd=10 1 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 20 client.4437.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 10 client.4437.objecter successfully canceled tick 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f73ac078e50 msgr2=0x7f73ac07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f73ac078e50 0x7f73ac07b310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73dc1566b0 tx=0x7f73c8039040 comp rx=0 tx=0).stop 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 10 monclient: shutdown 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 msgr2=0x7f73dc145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 --2- 10.20.193.51:0/984994084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f73dc15aec0 0x7f73dc145db0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73d000efc0 tx=0x7f73d000c490 comp rx=0 tx=0).stop 2026-02-22T21:37:38.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73dbfff640 1 -- 10.20.193.51:0/984994084 reap_dead start 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 shutdown_connections 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 >> 10.20.193.51:0/984994084 conn(0x7f73dc0ce4b0 msgr2=0x7f73dc15b350 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 shutdown_connections 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 -- 10.20.193.51:0/984994084 wait complete. 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.344+0000 7f73e1f5c640 1 librados: shutdown 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0006e60 con 0x7f4cc4152ab0 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: dump: 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc0010070 con 0x7f4cc4152ab0 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc000ca20 con 0x7f4cc4152ab0 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 5 monclient: authenticate success, global_id 4468 2026-02-22T21:37:38.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 monclient: _renew_subs 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb3fff640 4 set_mon_vals no callback set 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc4153970 con 0x7f4cc4152ab0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4cb27fc640 10 monclient: dump: 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 monclient: _renew_subs 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc4153f30 con 0x7f4cc4152ab0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 1 librados: init done 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a54082e30 con 0x7f6a640c9020 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f6a617fa640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 client.4468.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 15 client.4468.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 20 client.4468.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4cc41fb090 con 0x7f4cc4152ab0 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30017070 con 0x7f7c4015c790 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: dump: 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3000abe0 con 0x7f7c4015c790 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c30012400 con 0x7f7c4015c790 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 5 monclient: authenticate success, global_id 4471 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 10 monclient: _renew_subs 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c3d7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c40152160 con 0x7f7c4015c790 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c1f7fe640 10 monclient: dump: 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.414 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 10 monclient: _renew_subs 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c40152720 con 0x7f7c4015c790 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.345+0000 7f7c45643640 1 librados: init done 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f6a617fa640 10 client.4439.objecter ms_handle_connect 0x7f6a3c0813f0 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f6a69553640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f6a3c0813f0 0x7f6a3c083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 10 client.4471.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 15 client.4471.objecter _session_command_op_assign -1 1 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 20 client.4471.objecter _get_session osd=-1 returning homeless 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.346+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c40151c30 con 0x7f7c4015c790 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 10 client.4465.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 15 client.4465.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 20 client.4465.objecter _get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 20 client.4465.objecter _get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 15 client.4465.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 20 client.4465.objecter get_session s=0x7f280c080cc0 osd=12 3 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 15 client.4465.objecter _session_command_op_assign 12 2 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 4 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 10 client.4465.objecter _send_command 2 2026-02-22T21:37:38.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.349+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 --> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f283c071b20 con 0x7f280c081390 2026-02-22T21:37:38.416 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.416 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-22T21:37:38.416 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.416 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2026-02-22T21:37:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.352+0000 7f2817fff640 1 -- 10.20.193.51:0/1052472058 <== osd.12 v2:10.20.193.51:6818/1676524147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f283c071b20 con 0x7f280c081390 2026-02-22T21:37:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.352+0000 7f2817fff640 10 client.4465.objecter ms_dispatch2 0x7f283c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.352+0000 7f2817fff640 10 client.4465.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.352+0000 7f2817fff640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 4 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.352+0000 7f2817fff640 15 client.4465.objecter _session_command_op_remove 12 2 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 10 librados: watch_flush enter 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 10 librados: watch_flush exit 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 10 client.4465.objecter close_session for osd.12 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f280c081390 msgr2=0x7f280c0837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f280c081390 0x7f280c0837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 20 client.4465.objecter put_session s=0x7f280c080cc0 osd=12 1 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 20 client.4465.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 10 client.4465.objecter successfully canceled tick 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f280c078e50 msgr2=0x7f280c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f280c078e50 0x7f280c07b310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f28280096b0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 10 monclient: shutdown 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 msgr2=0x7f283c1f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 --2- 10.20.193.51:0/1052472058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f283c155ec0 0x7f283c1f5640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f282c00b570 tx=0x7f282c00ba40 comp rx=0 tx=0).stop 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f283bfff640 1 -- 10.20.193.51:0/1052472058 reap_dead start 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 shutdown_connections 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 >> 10.20.193.51:0/1052472058 conn(0x7f283c0cc080 msgr2=0x7f283c1ea670 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 shutdown_connections 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 -- 10.20.193.51:0/1052472058 wait complete. 2026-02-22T21:37:38.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7f2841b40640 1 librados: shutdown 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.353+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd53003f030 con 0x7fd53c152790 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.354+0000 7fd529ffb640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd50c078e00 0x7fd50c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.354+0000 7fd5411de640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd50c078e00 0x7fd50c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.354+0000 7fd5411de640 10 monclient: get_auth_request con 0x7fd50c078e00 auth_method 0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.354+0000 7fd529ffb640 10 client.4459.objecter ms_handle_connect 0x7fd50c078e00 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.354+0000 7fd5411de640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd50c078e00 0x7fd50c07b2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd52c005980 tx=0x7fd52c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.355+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== osd.13 v2:10.20.193.68:6800/762840177 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fd55c083f00 con 0x7fd55c0813f0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.355+0000 7fd588ff9640 10 client.4449.objecter ms_dispatch2 0x7fd58c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.355+0000 7fd588ff9640 10 client.4449.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.355+0000 7fd588ff9640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 4 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.355+0000 7fd588ff9640 15 client.4449.objecter _session_command_op_remove 13 1 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.356+0000 7fc1777fe640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7fc148081390 0x7fc1480837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fd53002fa30 con 0x7fd53c152790 2026-02-22T21:37:38.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 10 client.4459.objecter ms_dispatch2 0x7fd53c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 3 client.4459.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 3 client.4459.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 10 client.4459.objecter checking command 1 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7fd50c0813a0 0x7fd50c0837e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter _get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 15 client.4459.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter _get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 15 client.4459.objecter _session_command_op_assign 1 1 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 4 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 10 client.4459.objecter _send_command 1 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd50c083eb0 con 0x7fd50c0813a0 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd529ffb640 20 client.4459.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.359+0000 7fd5419df640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7fd50c0813a0 0x7fd50c0837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.360+0000 7fd5419df640 10 monclient: get_auth_request con 0x7fd50c0813a0 auth_method 0 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.360+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd5300836d0 con 0x7fd53c152790 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.360+0000 7fd529ffb640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stdout:103079215109 2026-02-22T21:37:38.419 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2026-02-22T21:37:38.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd5419df640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7fd50c0813a0 0x7fd50c0837e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd529ffb640 10 client.4459.objecter ms_handle_connect 0x7fd50c0813a0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f2e6c00b6b0 con 0x7f2e841f3700 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 10 client.4449.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 15 client.4449.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 20 client.4449.objecter _get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 20 client.4449.objecter _get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 15 client.4449.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 20 client.4449.objecter get_session s=0x7fd55c080d20 osd=13 3 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 15 client.4449.objecter _session_command_op_assign 13 2 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 4 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 10 client.4449.objecter _send_command 2 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd58c1f94d0 con 0x7fd55c0813f0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.362+0000 7f2e637fe640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2e50078e80 0x7f2e5007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.363+0000 7f2e82ffd640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2e50078e80 0x7f2e5007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.363+0000 7f2e82ffd640 10 monclient: get_auth_request con 0x7f2e50078e80 auth_method 0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.363+0000 7f2e82ffd640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2e50078e80 0x7f2e5007b340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e78002930 tx=0x7f2e78007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.363+0000 7f2e637fe640 10 client.4452.objecter ms_handle_connect 0x7f2e50078e80 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.365+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== osd.0 v2:10.20.193.51:6802/1198482343 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fbdfc018e50 con 0x7fbddc081390 2026-02-22T21:37:38.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.365+0000 7fbe08ff9640 10 client.4436.objecter ms_dispatch2 0x7fbe0c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.365+0000 7fbe08ff9640 10 client.4436.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.365+0000 7fbe08ff9640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 4 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.365+0000 7fbe08ff9640 15 client.4436.objecter _session_command_op_remove 0 1 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd588ff9640 1 -- 10.20.193.51:0/3368503904 <== osd.13 v2:10.20.193.68:6800/762840177 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd58c1f94d0 con 0x7fd55c0813f0 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd588ff9640 10 client.4449.objecter ms_dispatch2 0x7fd58c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd588ff9640 10 client.4449.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd588ff9640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 4 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd588ff9640 15 client.4449.objecter _session_command_op_remove 13 2 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 10 librados: watch_flush enter 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 10 librados: watch_flush exit 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 10 client.4449.objecter close_session for osd.13 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fd55c0813f0 msgr2=0x7fd55c083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fd55c0813f0 0x7fd55c083830 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 20 client.4449.objecter put_session s=0x7fd55c080d20 osd=13 1 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 20 client.4449.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 10 client.4449.objecter successfully canceled tick 2026-02-22T21:37:38.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd55c078e50 msgr2=0x7fd55c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd55c078e50 0x7fd55c07b310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd578009750 tx=0x7fd578006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 10 monclient: shutdown 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 msgr2=0x7fd58c1fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 --2- 10.20.193.51:0/3368503904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd58c1526a0 0x7fd58c1fbe10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00b500 tx=0x7fd57c00b9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd590891640 1 -- 10.20.193.51:0/3368503904 reap_dead start 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 shutdown_connections 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 >> 10.20.193.51:0/3368503904 conn(0x7fd58c0ce350 msgr2=0x7fd58c0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 shutdown_connections 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 -- 10.20.193.51:0/3368503904 wait complete. 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.366+0000 7fd59231b640 1 librados: shutdown 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== osd.9 v2:10.20.193.68:6808/1841964582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f6a54043e50 con 0x7f6a3c0813f0 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f6a617fa640 10 client.4439.objecter ms_dispatch2 0x7f6a64000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f6a617fa640 10 client.4439.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f6a617fa640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 4 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f6a617fa640 15 client.4439.objecter _session_command_op_remove 9 1 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4cc001d030 con 0x7f4cc4152ab0 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.369+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f2fd4003a80 con 0x7f2fe4143c90 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f4cb27fc640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4c94078e50 0x7f4c9407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f4cc990b640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4c94078e50 0x7f4c9407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f4cc990b640 10 monclient: get_auth_request con 0x7f4c94078e50 auth_method 0 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f2fd1ffb640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fb4078e50 0x7f2fb407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f2fe8efa640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fb4078e50 0x7f2fb407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f2fe8efa640 10 monclient: get_auth_request con 0x7f2fb4078e50 auth_method 0 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f4cb27fc640 10 client.4468.objecter ms_handle_connect 0x7f4c94078e50 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f4cc990b640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4c94078e50 0x7f4c9407b310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40c4d10 tx=0x7f4cb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f2fd1ffb640 10 client.4458.objecter ms_handle_connect 0x7f2fb4078e50 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.370+0000 7f2fe8efa640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fb4078e50 0x7f2fb407b310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fe414e340 tx=0x7f2fd8002d00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 10 client.4436.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 15 client.4436.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 20 client.4436.objecter _get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 20 client.4436.objecter _get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 15 client.4436.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 20 client.4436.objecter get_session s=0x7fbddc080cc0 osd=0 3 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 15 client.4436.objecter _session_command_op_assign 0 2 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 4 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 10 client.4436.objecter _send_command 2 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.372+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 --> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe0c15edd0 con 0x7fbddc081390 2026-02-22T21:37:38.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.373+0000 7fc1777fe640 10 monclient: get_auth_request con 0x7fc148081390 auth_method 0 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:38.424 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7fbe08ff9640 1 -- 10.20.193.51:0/2719535202 <== osd.0 v2:10.20.193.51:6802/1198482343 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbe0c15edd0 con 0x7fbddc081390 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7fbe08ff9640 10 client.4436.objecter ms_dispatch2 0x7fbe0c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7fbe08ff9640 10 client.4436.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7fbe08ff9640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 4 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7fbe08ff9640 15 client.4436.objecter _session_command_op_remove 0 2 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f4cc005d850 con 0x7f4cc4152ab0 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7f4cb27fc640 10 client.4468.objecter ms_dispatch2 0x7f4cc4000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7f4cb27fc640 3 client.4468.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.375+0000 7f4cb27fc640 3 client.4468.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 10 client.4468.objecter checking command 1 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f4c94081390 0x7f4c940837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter _get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 15 client.4468.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter _get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 15 client.4468.objecter _session_command_op_assign 11 1 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 4 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 10 client.4468.objecter _send_command 1 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 10 librados: watch_flush enter 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cc000ee50 con 0x7f4c94081390 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 20 client.4468.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 10 librados: watch_flush exit 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 10 client.4436.objecter close_session for osd.0 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7fbddc081390 msgr2=0x7fbddc0837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7fbddc081390 0x7fbddc0837d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 20 client.4436.objecter put_session s=0x7fbddc080cc0 osd=0 1 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 20 client.4436.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 10 client.4436.objecter successfully canceled tick 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbddc078e50 msgr2=0x7fbddc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbddc078e50 0x7fbddc07b310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c15f410 tx=0x7fbdf8031000 comp rx=0 tx=0).stop 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 10 monclient: shutdown 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 msgr2=0x7fbe0c148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 --2- 10.20.193.51:0/2719535202 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe0c152a80 0x7fbe0c148c70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00b5a0 tx=0x7fbdfc00ba70 comp rx=0 tx=0).stop 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe10ed2640 1 -- 10.20.193.51:0/2719535202 reap_dead start 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 shutdown_connections 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 >> 10.20.193.51:0/2719535202 conn(0x7fbe0c0ce350 msgr2=0x7fbe0c0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 10 client.4439.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 15 client.4439.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 20 client.4439.objecter _get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 20 client.4439.objecter _get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 15 client.4439.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 20 client.4439.objecter get_session s=0x7f6a3c080d20 osd=9 3 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 15 client.4439.objecter _session_command_op_assign 9 2 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 4 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 10 client.4439.objecter _send_command 2 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 --> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a64073d10 con 0x7f6a3c0813f0 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 shutdown_connections 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 -- 10.20.193.51:0/2719535202 wait complete. 2026-02-22T21:37:38.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7fbe1295c640 1 librados: shutdown 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cc910a640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f4c94081390 0x7f4c940837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cc910a640 10 monclient: get_auth_request con 0x7f4c94081390 auth_method 0 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4cc005dc10 con 0x7f4cc4152ab0 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.376+0000 7f4cb27fc640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.378+0000 7f4cc910a640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f4c94081390 0x7f4c940837d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.378+0000 7f4cb27fc640 10 client.4468.objecter ms_handle_connect 0x7f4c94081390 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f2e6c05e400 con 0x7f2e841f3700 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 10 client.4452.objecter ms_dispatch2 0x7f2e84000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 3 client.4452.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 3 client.4452.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 10 client.4452.objecter checking command 1 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7f2e50081420 0x7f2e50083860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter _get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 15 client.4452.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter _get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 15 client.4452.objecter _session_command_op_assign 4 1 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 4 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 10 client.4452.objecter _send_command 1 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e6c00cfa0 con 0x7f2e50081420 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 20 client.4452.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2e6c05e7c0 con 0x7f2e841f3700 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e637fe640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f2fd4024e70 con 0x7f2fe4143c90 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 10 client.4458.objecter ms_dispatch2 0x7f2fe4000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 3 client.4458.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 3 client.4458.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 10 client.4458.objecter checking command 1 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f2fb40813f0 0x7f2fb4083830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter _get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 15 client.4458.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter _get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 15 client.4458.objecter _session_command_op_assign 2 1 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2e837fe640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7f2e50081420 0x7f2e50083860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 4 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 10 client.4458.objecter _send_command 1 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fd4014e50 con 0x7f2fb40813f0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 20 client.4458.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fd405e410 con 0x7f2fe4143c90 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.379+0000 7f2fd1ffb640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.380+0000 7f2e837fe640 10 monclient: get_auth_request con 0x7f2e50081420 auth_method 0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.380+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f2fb40813f0 0x7f2fb4083830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.380+0000 7f2fe9efc640 10 monclient: get_auth_request con 0x7f2fb40813f0 auth_method 0 2026-02-22T21:37:38.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.382+0000 7f2e637fe640 10 client.4452.objecter ms_handle_connect 0x7f2e50081420 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.382+0000 7f2e837fe640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7f2e50081420 0x7f2e50083860 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.382+0000 7f2fd1ffb640 10 client.4458.objecter ms_handle_connect 0x7f2fb40813f0 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.382+0000 7f2fe9efc640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f2fb40813f0 0x7f2fb4083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a617fa640 1 -- 10.20.193.51:0/1236403597 <== osd.9 v2:10.20.193.68:6808/1841964582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6a64073d10 con 0x7f6a3c0813f0 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a617fa640 10 client.4439.objecter ms_dispatch2 0x7f6a64000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a617fa640 10 client.4439.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a617fa640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 4 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a617fa640 15 client.4439.objecter _session_command_op_remove 9 2 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 10 librados: watch_flush enter 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 10 librados: watch_flush exit 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 10 client.4439.objecter close_session for osd.9 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7c3001b3b0 con 0x7f7c4015c790 2026-02-22T21:37:38.429 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.430 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f6a3c0813f0 msgr2=0x7f6a3c083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f6a3c0813f0 0x7f6a3c083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 20 client.4439.objecter put_session s=0x7f6a3c080d20 osd=9 1 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 20 client.4439.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 10 client.4439.objecter successfully canceled tick 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a3c078e50 msgr2=0x7f6a3c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6a3c078e50 0x7f6a3c07b310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006b90 tx=0x7f6a50008040 comp rx=0 tx=0).stop 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 10 monclient: shutdown 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 msgr2=0x7f6a641f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 --2- 10.20.193.51:0/1236403597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6a640c9020 0x7f6a641f1630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009a70 tx=0x7f6a540047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69553640 1 -- 10.20.193.51:0/1236403597 reap_dead start 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 shutdown_connections 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 >> 10.20.193.51:0/1236403597 conn(0x7f6a640ce340 msgr2=0x7f6a640d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 shutdown_connections 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 -- 10.20.193.51:0/1236403597 wait complete. 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.384+0000 7f6a69d54640 1 librados: shutdown 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== osd.1 v2:10.20.193.68:6824/2540990948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fd50c083eb0 con 0x7fd50c0813a0 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fd529ffb640 10 client.4459.objecter ms_dispatch2 0x7fd53c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fd529ffb640 10 client.4459.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fd529ffb640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 4 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fd529ffb640 15 client.4459.objecter _session_command_op_remove 1 1 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7f7c1f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7c0c078e80 0x7f7c0c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7f7c3effd640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7c0c078e80 0x7f7c0c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7f7c3effd640 10 monclient: get_auth_request con 0x7f7c0c078e80 auth_method 0 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7f7c3effd640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7c0c078e80 0x7f7c0c07b340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c40153300 tx=0x7f7c34007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7f7c1f7fe640 10 client.4471.objecter ms_handle_connect 0x7f7c0c078e80 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fc1777fe640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7fc148081390 0x7fc1480837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.385+0000 7fc14ffff640 10 client.4456.objecter ms_handle_connect 0x7fc148081390 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f7c3005deb0 con 0x7f7c4015c790 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 10 client.4471.objecter ms_dispatch2 0x7f7c40000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 3 client.4471.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 3 client.4471.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 10 client.4471.objecter checking command 1 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f7c0c0813c0 0x7f7c0c083800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter _get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 15 client.4471.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter _get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 15 client.4471.objecter _session_command_op_assign 7 1 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 4 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 10 client.4471.objecter _send_command 1 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c0c083ed0 con 0x7f7c0c0813c0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 20 client.4471.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7c3005e270 con 0x7f7c4015c790 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7f7c1f7fe640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb67003f050 con 0x7fb67c1526a0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.387+0000 7fb669ffb640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb64c078e50 0x7fb64c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.388+0000 7fb68113f640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb64c078e50 0x7fb64c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.388+0000 7fb68113f640 10 monclient: get_auth_request con 0x7fb64c078e50 auth_method 0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.388+0000 7fb68113f640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb64c078e50 0x7fb64c07b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00a830 tx=0x7fb66c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.388+0000 7fb669ffb640 10 client.4455.objecter ms_handle_connect 0x7fb64c078e50 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.389+0000 7f7c3e7fc640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f7c0c0813c0 0x7f7c0c083800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.389+0000 7f7c3e7fc640 10 monclient: get_auth_request con 0x7f7c0c0813c0 auth_method 0 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 10 client.4459.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 15 client.4459.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 20 client.4459.objecter _get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 20 client.4459.objecter _get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 15 client.4459.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 20 client.4459.objecter get_session s=0x7fd50c080cd0 osd=1 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 15 client.4459.objecter _session_command_op_assign 1 2 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 4 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 10 client.4459.objecter _send_command 2 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.391+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 --> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd53c15ce70 con 0x7fd50c0813a0 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fb670082fa0 con 0x7fb67c1526a0 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 10 client.4455.objecter ms_dispatch2 0x7fb67c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 3 client.4455.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 10 client.4455.objecter checking command 1 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7fb64c081390 0x7fb64c0837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter _get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 15 client.4455.objecter _session_command_op_remove -1 1 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter _get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 15 client.4455.objecter _session_command_op_assign 15 1 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 4 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 10 client.4455.objecter _send_command 1 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb64c083ea0 con 0x7fb64c081390 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 20 client.4455.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7f7c3e7fc640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f7c0c0813c0 0x7f7c0c083800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7f7c1f7fe640 10 client.4471.objecter ms_handle_connect 0x7f7c0c0813c0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb670083360 con 0x7fb67c1526a0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb669ffb640 10 monclient: handle_get_version_reply finishing 1 version 30 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.392+0000 7fb681940640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7fb64c081390 0x7fb64c0837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:38.434 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.393+0000 7fb681940640 10 monclient: get_auth_request con 0x7fb64c081390 auth_method 0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fb669ffb640 10 client.4455.objecter ms_handle_connect 0x7fb64c081390 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fb681940640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7fb64c081390 0x7fb64c0837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.15 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fd529ffb640 1 -- 10.20.193.51:0/1242726476 <== osd.1 v2:10.20.193.68:6824/2540990948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd53c15ce70 con 0x7fd50c0813a0 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fd529ffb640 10 client.4459.objecter ms_dispatch2 0x7fd53c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fd529ffb640 10 client.4459.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fd529ffb640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 4 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.394+0000 7fd529ffb640 15 client.4459.objecter _session_command_op_remove 1 2 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 10 librados: watch_flush enter 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 10 librados: watch_flush exit 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 10 client.4459.objecter close_session for osd.1 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7fd50c0813a0 msgr2=0x7fd50c0837e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7fd50c0813a0 0x7fd50c0837e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 20 client.4459.objecter put_session s=0x7fd50c080cd0 osd=1 1 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 20 client.4459.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 10 client.4459.objecter successfully canceled tick 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd50c078e00 msgr2=0x7fd50c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd50c078e00 0x7fd50c07b2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd52c005980 tx=0x7fd52c009040 comp rx=0 tx=0).stop 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 10 monclient: shutdown 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152790 msgr2=0x7fd53c0c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 --2- 10.20.193.51:0/1242726476 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd53c152790 0x7fd53c0c80a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd530036000 tx=0x7fd5300047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd5419df640 1 -- 10.20.193.51:0/1242726476 reap_dead start 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 shutdown_connections 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 >> 10.20.193.51:0/1242726476 conn(0x7fd53c06d9c0 msgr2=0x7fd53c14ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 shutdown_connections 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 -- 10.20.193.51:0/1242726476 wait complete. 2026-02-22T21:37:38.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.395+0000 7fd543469640 1 librados: shutdown 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.399+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== osd.11 v2:10.20.193.140:6816/2298497951 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f4cc000ee50 con 0x7f4c94081390 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.399+0000 7f4cb27fc640 10 client.4468.objecter ms_dispatch2 0x7f4cc4000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.399+0000 7f4cb27fc640 10 client.4468.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.399+0000 7f4cb27fc640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 4 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.399+0000 7f4cb27fc640 15 client.4468.objecter _session_command_op_remove 11 1 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.402+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== osd.2 v2:10.20.193.125:6808/1963822926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f2fd4014e50 con 0x7f2fb40813f0 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.402+0000 7f2fd1ffb640 10 client.4458.objecter ms_dispatch2 0x7f2fe4000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.402+0000 7f2fd1ffb640 10 client.4458.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.402+0000 7f2fd1ffb640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 4 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.402+0000 7f2fd1ffb640 15 client.4458.objecter _session_command_op_remove 2 1 2026-02-22T21:37:38.436 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:38.436 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.404+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== osd.4 v2:10.20.193.51:6810/1021106712 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f2e6c00cfa0 con 0x7f2e50081420 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.404+0000 7f2e637fe640 10 client.4452.objecter ms_dispatch2 0x7f2e84000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.404+0000 7f2e637fe640 10 client.4452.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.404+0000 7f2e637fe640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 4 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.404+0000 7f2e637fe640 15 client.4452.objecter _session_command_op_remove 4 1 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 10 client.4468.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 15 client.4468.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 20 client.4468.objecter _get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 20 client.4468.objecter _get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 15 client.4468.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 20 client.4468.objecter get_session s=0x7f4c94080cc0 osd=11 3 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 15 client.4468.objecter _session_command_op_assign 11 2 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 4 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 10 client.4468.objecter _send_command 2 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 --> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4cc40c5ab0 con 0x7f4c94081390 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== osd.8 v2:10.20.193.51:6826/142198174 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fc16001fe70 con 0x7fc148081390 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7fc14ffff640 10 client.4456.objecter ms_dispatch2 0x7fc178000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7fc14ffff640 10 client.4456.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7fc14ffff640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 4 2026-02-22T21:37:38.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.407+0000 7fc14ffff640 15 client.4456.objecter _session_command_op_remove 8 1 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 10 client.4458.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 15 client.4458.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 20 client.4458.objecter _get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 20 client.4458.objecter _get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 15 client.4458.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 20 client.4458.objecter get_session s=0x7f2fb4080d20 osd=2 3 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 15 client.4458.objecter _session_command_op_assign 2 2 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 4 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 10 client.4458.objecter _send_command 2 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.409+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 --> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2fe4150280 con 0x7f2fb40813f0 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.410+0000 7f4cb27fc640 1 -- 10.20.193.51:0/3649964800 <== osd.11 v2:10.20.193.140:6816/2298497951 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4cc40c5ab0 con 0x7f4c94081390 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.410+0000 7f4cb27fc640 10 client.4468.objecter ms_dispatch2 0x7f4cc4000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.410+0000 7f4cb27fc640 10 client.4468.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.410+0000 7f4cb27fc640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 4 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.410+0000 7f4cb27fc640 15 client.4468.objecter _session_command_op_remove 11 2 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 10 librados: watch_flush enter 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 10 librados: watch_flush exit 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 10 client.4468.objecter close_session for osd.11 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f4c94081390 msgr2=0x7f4c940837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f4c94081390 0x7f4c940837d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 20 client.4468.objecter put_session s=0x7f4c94080cc0 osd=11 1 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 20 client.4468.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 10 client.4468.objecter successfully canceled tick 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4c94078e50 msgr2=0x7f4c9407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4c94078e50 0x7f4c9407b310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40c4d10 tx=0x7f4cb4008040 comp rx=0 tx=0).stop 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 10 monclient: shutdown 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 msgr2=0x7f4cc41fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 --2- 10.20.193.51:0/3649964800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4cc4152ab0 0x7f4cc41fb2b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4cc000ba10 tx=0x7f4cc000bee0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4cca10c640 1 -- 10.20.193.51:0/3649964800 reap_dead start 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 shutdown_connections 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 >> 10.20.193.51:0/3649964800 conn(0x7f4cc40cd7b0 msgr2=0x7f4cc40c0480 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 shutdown_connections 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 -- 10.20.193.51:0/3649964800 wait complete. 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f4ccbb96640 1 librados: shutdown 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 10 client.4452.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 15 client.4452.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 20 client.4452.objecter _get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 20 client.4452.objecter _get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 15 client.4452.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 20 client.4452.objecter get_session s=0x7f2e50080d50 osd=4 3 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 15 client.4452.objecter _session_command_op_assign 4 2 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 4 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 10 client.4452.objecter _send_command 2 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.411+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 --> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e8414ec30 con 0x7f2e50081420 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2fd1ffb640 1 -- 10.20.193.51:0/192091311 <== osd.2 v2:10.20.193.125:6808/1963822926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2fe4150280 con 0x7f2fb40813f0 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2fd1ffb640 10 client.4458.objecter ms_dispatch2 0x7f2fe4000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2fd1ffb640 10 client.4458.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2fd1ffb640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 4 2026-02-22T21:37:38.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2fd1ffb640 15 client.4458.objecter _session_command_op_remove 2 2 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 10 librados: watch_flush enter 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 10 librados: watch_flush exit 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 10 client.4458.objecter close_session for osd.2 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f2fb40813f0 msgr2=0x7f2fb4083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f2fb40813f0 0x7f2fb4083830 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 20 client.4458.objecter put_session s=0x7f2fb4080d20 osd=2 1 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 20 client.4458.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 10 client.4458.objecter successfully canceled tick 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fb4078e50 msgr2=0x7f2fb407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fb4078e50 0x7f2fb407b310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fe414e340 tx=0x7f2fd8002d00 comp rx=0 tx=0).stop 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.412+0000 7f2feb986640 10 monclient: shutdown 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 msgr2=0x7f2fe41496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 --2- 10.20.193.51:0/192091311 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2fe4143c90 0x7f2fe41496d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c760 tx=0x7f2fd400cc30 comp rx=0 tx=0).stop 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2fe9efc640 1 -- 10.20.193.51:0/192091311 reap_dead start 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 shutdown_connections 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 >> 10.20.193.51:0/192091311 conn(0x7f2fe40ce350 msgr2=0x7f2fe40d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 shutdown_connections 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 -- 10.20.193.51:0/192091311 wait complete. 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f2feb986640 1 librados: shutdown 2026-02-22T21:37:38.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== osd.7 v2:10.20.193.140:6808/1140578699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f7c0c083ed0 con 0x7f7c0c0813c0 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f7c1f7fe640 10 client.4471.objecter ms_dispatch2 0x7f7c40000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f7c1f7fe640 10 client.4471.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f7c1f7fe640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 4 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.413+0000 7f7c1f7fe640 15 client.4471.objecter _session_command_op_remove 7 1 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 10 client.4456.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 15 client.4456.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 20 client.4456.objecter _get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 20 client.4456.objecter _get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 15 client.4456.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 20 client.4456.objecter get_session s=0x7fc148080cc0 osd=8 3 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 15 client.4456.objecter _session_command_op_assign 8 2 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 4 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 10 client.4456.objecter _send_command 2 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.414+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 --> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc1780c8470 con 0x7fc148081390 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e637fe640 1 -- 10.20.193.51:0/2582322711 <== osd.4 v2:10.20.193.51:6810/1021106712 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2e8414ec30 con 0x7f2e50081420 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e637fe640 10 client.4452.objecter ms_dispatch2 0x7f2e84000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e637fe640 10 client.4452.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e637fe640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 4 2026-02-22T21:37:38.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e637fe640 15 client.4452.objecter _session_command_op_remove 4 2 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 10 librados: watch_flush enter 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 10 librados: watch_flush exit 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 10 client.4452.objecter close_session for osd.4 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7f2e50081420 msgr2=0x7f2e50083860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7f2e50081420 0x7f2e50083860 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 20 client.4452.objecter put_session s=0x7f2e50080d50 osd=4 1 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 20 client.4452.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 10 client.4452.objecter successfully canceled tick 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2e50078e80 msgr2=0x7f2e5007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2e50078e80 0x7f2e5007b340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e78002930 tx=0x7f2e78007e40 comp rx=0 tx=0).stop 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 10 monclient: shutdown 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 msgr2=0x7f2e840d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 --2- 10.20.193.51:0/2582322711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2e841f3700 0x7f2e840d0920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00d7a0 tx=0x7f2e6c00dc70 comp rx=0 tx=0).stop 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e837fe640 1 -- 10.20.193.51:0/2582322711 reap_dead start 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 shutdown_connections 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 >> 10.20.193.51:0/2582322711 conn(0x7f2e8406d8c0 msgr2=0x7f2e840c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 shutdown_connections 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 -- 10.20.193.51:0/2582322711 wait complete. 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.415+0000 7f2e897ce640 1 librados: shutdown 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc14ffff640 1 -- 10.20.193.51:0/4121244979 <== osd.8 v2:10.20.193.51:6826/142198174 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc1780c8470 con 0x7fc148081390 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc14ffff640 10 client.4456.objecter ms_dispatch2 0x7fc178000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc14ffff640 10 client.4456.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc14ffff640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 4 2026-02-22T21:37:38.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc14ffff640 15 client.4456.objecter _session_command_op_remove 8 2 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 10 librados: watch_flush enter 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 10 librados: watch_flush exit 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 10 client.4456.objecter close_session for osd.8 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7fc148081390 msgr2=0x7fc1480837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7fc148081390 0x7fc1480837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 20 client.4456.objecter put_session s=0x7fc148080cc0 osd=8 1 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 20 client.4456.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 10 client.4456.objecter successfully canceled tick 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc148078e50 msgr2=0x7fc14807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc148078e50 0x7fc14807b310 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc16c02eef0 tx=0x7fc16c033000 comp rx=0 tx=0).stop 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 10 monclient: shutdown 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc178153970 msgr2=0x7fc1780bff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 --2- 10.20.193.51:0/4121244979 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc178153970 0x7fc1780bff50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc16000e870 tx=0x7fc16000ed40 comp rx=0 tx=0).stop 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc1777fe640 1 -- 10.20.193.51:0/4121244979 reap_dead start 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 shutdown_connections 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 >> 10.20.193.51:0/4121244979 conn(0x7fc1780ca240 msgr2=0x7fc178159820 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 shutdown_connections 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 -- 10.20.193.51:0/4121244979 wait complete. 2026-02-22T21:37:38.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.418+0000 7fc17d944640 1 librados: shutdown 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stdout:94489280517 2026-02-22T21:37:38.444 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.419+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== osd.15 v2:10.20.193.140:6800/2657947218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fb64c083ea0 con 0x7fb64c081390 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.419+0000 7fb669ffb640 10 client.4455.objecter ms_dispatch2 0x7fb67c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.419+0000 7fb669ffb640 10 client.4455.objecter _finish_command 1 = system:0 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.419+0000 7fb669ffb640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 4 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.419+0000 7fb669ffb640 15 client.4455.objecter _session_command_op_remove 15 1 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 10 client.4471.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 15 client.4471.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 20 client.4471.objecter _get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 20 client.4471.objecter _get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 15 client.4471.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 20 client.4471.objecter get_session s=0x7f7c0c080cf0 osd=7 3 2026-02-22T21:37:38.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 15 client.4471.objecter _session_command_op_assign 7 2 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 4 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 10 client.4471.objecter _send_command 2 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.420+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 --> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c400cd2b0 con 0x7f7c0c0813c0 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:38.445 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c1f7fe640 1 -- 10.20.193.51:0/1932056164 <== osd.7 v2:10.20.193.140:6808/1140578699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7c400cd2b0 con 0x7f7c0c0813c0 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c1f7fe640 10 client.4471.objecter ms_dispatch2 0x7f7c40000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c1f7fe640 10 client.4471.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c1f7fe640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 4 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c1f7fe640 15 client.4471.objecter _session_command_op_remove 7 2 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 10 librados: watch_flush enter 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 10 librados: watch_flush exit 2026-02-22T21:37:38.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 10 client.4471.objecter close_session for osd.7 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f7c0c0813c0 msgr2=0x7f7c0c083800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f7c0c0813c0 0x7f7c0c083800 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 20 client.4471.objecter put_session s=0x7f7c0c080cf0 osd=7 1 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 20 client.4471.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 10 client.4471.objecter successfully canceled tick 2026-02-22T21:37:38.446 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7c0c078e80 msgr2=0x7f7c0c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7c0c078e80 0x7f7c0c07b340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c40153300 tx=0x7f7c34007ab0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 10 monclient: shutdown 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7c4015c790 msgr2=0x7f7c40154540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 --2- 10.20.193.51:0/1932056164 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7c4015c790 0x7f7c40154540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000caa0 tx=0x7f7c3000cf70 comp rx=0 tx=0).stop 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c3f7fe640 1 -- 10.20.193.51:0/1932056164 reap_dead start 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 shutdown_connections 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 >> 10.20.193.51:0/1932056164 conn(0x7f7c4006d8c0 msgr2=0x7f7c400c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 shutdown_connections 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 -- 10.20.193.51:0/1932056164 wait complete. 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.423+0000 7f7c45643640 1 librados: shutdown 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 10 client.4455.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 15 client.4455.objecter _session_command_op_assign -1 2 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 20 client.4455.objecter _get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 20 client.4455.objecter _get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 15 client.4455.objecter _session_command_op_remove -1 2 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 20 client.4455.objecter get_session s=0x7fb64c080cc0 osd=15 3 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 15 client.4455.objecter _session_command_op_assign 15 2 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 4 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 10 client.4455.objecter _send_command 2 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.427+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 --> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb67c14c4e0 con 0x7fb64c081390 2026-02-22T21:37:38.447 INFO:teuthology.orchestra.run.trial051.stdout:103079215109 2026-02-22T21:37:38.447 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb669ffb640 1 -- 10.20.193.51:0/3929518326 <== osd.15 v2:10.20.193.140:6800/2657947218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb67c14c4e0 con 0x7fb64c081390 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb669ffb640 10 client.4455.objecter ms_dispatch2 0x7fb67c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb669ffb640 10 client.4455.objecter _finish_command 2 = system:0 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb669ffb640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 4 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb669ffb640 15 client.4455.objecter _session_command_op_remove 15 2 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 10 librados: watch_flush enter 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 10 librados: watch_flush exit 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 10 client.4455.objecter close_session for osd.15 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7fb64c081390 msgr2=0x7fb64c0837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7fb64c081390 0x7fb64c0837d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 20 client.4455.objecter put_session s=0x7fb64c080cc0 osd=15 1 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 20 client.4455.objecter shutdown clearing up homeless session... 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 10 client.4455.objecter successfully canceled tick 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb64c078e50 msgr2=0x7fb64c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb64c078e50 0x7fb64c07b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00a830 tx=0x7fb66c008040 comp rx=0 tx=0).stop 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 10 monclient: shutdown 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 msgr2=0x7fb67c1e9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 --2- 10.20.193.51:0/3929518326 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb67c1526a0 0x7fb67c1e9af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb670009aa0 tx=0x7fb6700047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb681940640 1 -- 10.20.193.51:0/3929518326 reap_dead start 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stdout:94489280517 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 shutdown_connections 2026-02-22T21:37:38.449 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 >> 10.20.193.51:0/3929518326 conn(0x7fb67c0ce350 msgr2=0x7fb67c0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 shutdown_connections 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 -- 10.20.193.51:0/3929518326 wait complete. 2026-02-22T21:37:38.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.430+0000 7fb6833ca640 1 librados: shutdown 2026-02-22T21:37:38.450 INFO:teuthology.orchestra.run.trial051.stdout:90194313222 2026-02-22T21:37:38.450 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.504+0000 7f8a81fd5640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.504+0000 7f8a81fd5640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.504+0000 7f8a81fd5640 10 monclient: monmap: 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.505587+0000 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.505587+0000 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.506 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.505+0000 7f8a81fd5640 1 Processor -- start 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.505+0000 7f8a81fd5640 1 -- start start 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.505+0000 7f8a81fd5640 10 monclient: init 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient: _add_conns ranks=[2,3,1,5,0,4] 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a7c15b930 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f8a7c15b150 0x7f8a7c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7c15b150 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f8a7c156040 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7c156460 con 0x7f8a7c156040 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7c159dd0 con 0x7f8a7c15b930 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f8a7c15acc0 con 0x7f8a7c15b150 2026-02-22T21:37:38.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a81fd5640 10 monclient(hunting): authenticate will time out at 1134.086195s 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.506+0000 7f8a7affd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f8a7c15b150 0x7f8a7c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f8a7c15b150 0x7f8a7c15bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60876/0 (socket says 10.20.193.51:60876) 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 1 -- 10.20.193.51:0/1779050631 learned_addr learned my addr 10.20.193.51:0/1779050631 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): get_auth_request con 0x7f8a7c15b150 auth_method 0 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.507+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2026-02-22T21:37:38.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 1 -- 10.20.193.51:0/1779050631 >> v1:10.20.193.125:6789/0 conn(0x7f8a7c15b930 legacy=0x7f8a7c1595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 1 -- 10.20.193.51:0/1779050631 >> v1:10.20.193.68:6789/0 conn(0x7f8a7c156040 legacy=0x7f8a7c151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.509+0000 7f8a7affd640 1 -- 10.20.193.51:0/1779050631 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c15a0b0 con 0x7f8a7c15b150 2026-02-22T21:37:38.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.511+0000 7f8a7affd640 1 --2- 10.20.193.51:0/1779050631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c15bd10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c15ae40 tx=0x7f8a6c030130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc03adb1baf62018 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 1 -- 10.20.193.51:0/1779050631 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c03d070 con 0x7f8a7c15b150 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: dump: 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.516 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 1 -- 10.20.193.51:0/1779050631 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6c030d60 con 0x7f8a7c15b150 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 1 -- 10.20.193.51:0/1779050631 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c038660 con 0x7f8a7c15b150 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a7a7fc640 10 monclient: dump: 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.517 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a81fd5640 5 monclient: authenticate success, global_id 4461 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a81fd5640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.514+0000 7f8a81fd5640 4 set_mon_vals no callback set 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 10 monclient: shutdown 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1779050631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 msgr2=0x7f8a7c15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1779050631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c15bd10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c15ae40 tx=0x7f8a6c030130 comp rx=0 tx=0).stop 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1779050631 shutdown_connections 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1779050631 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c15bd10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1779050631 >> 10.20.193.51:0/1779050631 conn(0x7f8a7c0cd7b0 msgr2=0x7f8a7c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1779050631 shutdown_connections 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.515+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1779050631 wait complete. 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.516+0000 7f8a81fd5640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.516+0000 7f8a81fd5640 10 monclient: monmap: 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.519 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 librados: starting msgr at 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 librados: starting objecter 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 Processor -- start 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 -- start start 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 librados: setting wanted keys 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 1 librados: calling monclient init 2026-02-22T21:37:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.517+0000 7f8a81fd5640 10 monclient: init 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7c156040 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c1f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7c15b150 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7c15b930 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c15a950 con 0x7f8a7c15b150 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c15a670 con 0x7f8a7c156040 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c152190 con 0x7f8a7c15b930 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a81fd5640 10 monclient(hunting): authenticate will time out at 1134.098407s 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a7bfff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a7affd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c1f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.518+0000 7f8a7b7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35466/0 (socket says 10.20.193.51:35466) 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43738/0 (socket says 10.20.193.51:43738) 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 1 -- 10.20.193.51:0/1941222871 learned_addr learned my addr 10.20.193.51:0/1941222871 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f8a7c156040 auth_method 0 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): get_auth_request con 0x7f8a7c15b930 auth_method 0 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): get_auth_request con 0x7f8a7c15b150 auth_method 0 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.519+0000 7f8a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 -- 10.20.193.51:0/1941222871 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 msgr2=0x7f8a7c1efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 -- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 msgr2=0x7f8a7c1f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c1f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7b7fe640 1 -- 10.20.193.51:0/1941222871 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c1f0240 con 0x7f8a7c156040 2026-02-22T21:37:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.520+0000 7f8a7bfff640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f8a7b7fe640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800da80 tx=0x7f8a6800df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f8a57fff640 10 client.?.objecter ms_handle_connect 0x7f8a7c156040 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f8a57fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f53bc798640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f53bc798640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.521+0000 7f53bc798640 10 monclient: monmap: 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.522600+0000 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.522600+0000 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.525 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.522+0000 7f53bc798640 1 Processor -- start 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.522+0000 7f53bc798640 1 -- start start 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.522+0000 7f53bc798640 10 monclient: init 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 10 monclient: monmap: 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.523788+0000 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.523788+0000 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.527 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient: _add_conns ranks=[0,3,5,1,4,2] 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53b415b930 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f53b415b150 0x7f53b415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-a con 0x7f53b415b150 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f53b4156040 0x7f53b4151a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-c con 0x7f53b4156040 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b4156460 con 0x7f53b415b930 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f53b415acc0 con 0x7f53b415b150 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f53b4159dd0 con 0x7f53b4156040 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bc798640 10 monclient(hunting): authenticate will time out at 1134.103314s 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004070 con 0x7f8a7c156040 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: dump: 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.529 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68012070 con 0x7f8a7c156040 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a81fd5640 5 monclient: authenticate success, global_id 4474 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f53b4156040 0x7f53b4151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a81fd5640 10 monclient: _renew_subs 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a81fd5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7c0c2a90 con 0x7f8a7c156040 2026-02-22T21:37:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 1 Processor -- start 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a57fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a79ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 1 -- start start 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f9e20e58640 10 monclient: init 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f8a81fd5640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f53b4156040 0x7f53b4151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43750/0 (socket says 10.20.193.51:43750) 2026-02-22T21:37:38.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 1 -- 10.20.193.51:0/3224695755 learned_addr learned my addr 10.20.193.51:0/3224695755 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 10 monclient(hunting): get_auth_request con 0x7f53b4156040 auth_method 0 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.523+0000 7f53bad0e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004470 con 0x7f8a7c156040 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a57fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient: _add_conns ranks=[3,0,4,2,1,5] 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9e1c153d40 0x7f9e1c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1c153d40 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a57fff640 10 monclient: dump: 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.533 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a81fd5640 10 monclient: _renew_subs 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9915a68640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9915a68640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9915a68640 10 monclient: monmap: 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.525218+0000 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.525218+0000 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.534 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e1c154120 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9e1c0c8330 0x7f9e1c1645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1c0c8330 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c070230 con 0x7f9e1c154120 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c15d260 con 0x7f9e1c153d40 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1c070510 con 0x7f9e1c0c8330 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f9e20e58640 10 monclient(hunting): authenticate will time out at 1134.104665s 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a81fd5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7c0c3050 con 0x7f8a7c156040 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f8a81fd5640 1 librados: init done 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 1 -- 10.20.193.51:0/3224695755 >> v2:10.20.193.51:3300/0 conn(0x7f53b415b150 msgr2=0x7f53b415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 1 --2- 10.20.193.51:0/3224695755 >> v2:10.20.193.51:3300/0 conn(0x7f53b415b150 0x7f53b415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 1 -- 10.20.193.51:0/3224695755 >> v1:10.20.193.51:6789/0 conn(0x7f53b415b930 legacy=0x7f53b41595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.524+0000 7f53bad0e640 1 -- 10.20.193.51:0/3224695755 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b415a0b0 con 0x7f53b4156040 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f9915a68640 1 Processor -- start 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f9915a68640 1 -- start start 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f9915a68640 10 monclient: init 2026-02-22T21:37:38.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f8a81fd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f8a81fd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f8a81fd5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a7c15bd10 con 0x7f8a7c156040 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f53bad0e640 1 --2- 10.20.193.51:0/3224695755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b4156040 0x7f53b4151a80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53b4159cd0 tx=0x7f53b0031470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0220787b3b3351d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f9915a68640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f9915a68640 10 monclient: _add_conns ranks=[3,4,1,5,0,2] 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f99101526a0 0x7f99100733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-a con 0x7f99101526a0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9910152a80 0x7f99100cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-b con 0x7f9910152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f99100cdb20 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9910073d60 con 0x7f99100cdb20 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f99100cca00 con 0x7f99101526a0 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9910074c50 con 0x7f9910152a80 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9915a68640 10 monclient(hunting): authenticate will time out at 1134.106236s 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f42490c6640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f42490c6640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.525+0000 7f42490c6640 10 monclient: monmap: 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.526376+0000 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.526376+0000 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.539 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f42490c6640 1 Processor -- start 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f42490c6640 1 -- start start 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f42490c6640 10 monclient: init 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9e19d74640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f9e1c154120 0x7f9e1c0c7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46948/0 (socket says 10.20.193.51:46948) 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.526+0000 7f9e19d74640 1 -- 10.20.193.51:0/1905418106 learned_addr learned my addr 10.20.193.51:0/1905418106 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient: _add_conns ranks=[2,0,4,5,1,3] 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4244155e60 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f424415c790 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f42440c9470 0x7f42440d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-b con 0x7f42440c9470 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42440cbc80 con 0x7f424415c790 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 1 -- 10.20.193.51:0/3224695755 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0037070 con 0x7f53b4156040 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: dump: 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 1 -- 10.20.193.51:0/3224695755 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b003b600 con 0x7f53b4156040 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 1 -- 10.20.193.51:0/3224695755 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b003b9d0 con 0x7f53b4156040 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53b950b640 10 monclient: dump: 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53bc798640 5 monclient: authenticate success, global_id 4448 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53bc798640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f53bc798640 4 set_mon_vals no callback set 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42440cc450 con 0x7f4244155e60 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f42440cc200 con 0x7f42440c9470 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f42490c6640 10 monclient(hunting): authenticate will time out at 1134.107767s 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.527+0000 7f4242d76640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f4244155e60 0x7f42440cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33080/0 (socket says 10.20.193.51:33080) 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.528+0000 7f4242d76640 1 -- 10.20.193.51:0/3362931827 learned_addr learned my addr 10.20.193.51:0/3362931827 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.528+0000 7f53bc798640 10 monclient: shutdown 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.528+0000 7f53bc798640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.528+0000 7f53bc798640 1 -- 10.20.193.51:0/3224695755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b4156040 msgr2=0x7f53b4151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.528+0000 7f53bc798640 1 --2- 10.20.193.51:0/3224695755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b4156040 0x7f53b4151a80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53b4159cd0 tx=0x7f53b0031470 comp rx=0 tx=0).stop 2026-02-22T21:37:38.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 -- 10.20.193.51:0/3224695755 shutdown_connections 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 --2- 10.20.193.51:0/3224695755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b4156040 0x7f53b4151a80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 --2- 10.20.193.51:0/3224695755 >> v2:10.20.193.51:3300/0 conn(0x7f53b415b150 0x7f53b415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 -- 10.20.193.51:0/3224695755 >> 10.20.193.51:0/3224695755 conn(0x7f53b40cd7b0 msgr2=0x7f53b40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 -- 10.20.193.51:0/3224695755 shutdown_connections 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 1 -- 10.20.193.51:0/3224695755 wait complete. 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e1a575640 1 --2- 10.20.193.51:0/1905418106 >> v2:10.20.193.51:3300/0 conn(0x7f9e1c153d40 0x7f9e1c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 1 -- 10.20.193.51:0/1905418106 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131865643 0 0) 0x7f9e1c070230 con 0x7f9e1c154120 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 10 monclient(hunting): my global_id is 4467 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f9e19573640 1 -- 10.20.193.51:0/1905418106 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e08003620 con 0x7f9e1c154120 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220630734 0 0) 0x7f42440cc450 con 0x7f4244155e60 2026-02-22T21:37:38.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 10 monclient(hunting): my global_id is 4451 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f53bc798640 10 monclient: monmap: 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.529+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4230003620 con 0x7f4244155e60 2026-02-22T21:37:38.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 librados: starting msgr at 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 librados: starting objecter 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f9e1a575640 10 monclient(hunting): get_auth_request con 0x7f9e1c153d40 auth_method 0 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f9e1a575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f9e1a575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f9e1a575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 Processor -- start 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 -- start start 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 librados: setting wanted keys 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 1 librados: calling monclient init 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f53bc798640 10 monclient: init 2026-02-22T21:37:38.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.530+0000 7f990effd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f99101526a0 0x7f99100733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4156040 0x7f53b41fb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4156040 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b415b150 0x7f53b40737a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 10 monclient: monmap: 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.531882+0000 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.531882+0000 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.547 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-b con 0x7f53b415b150 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): picked mon.noname-c con 0x7f53b415b930 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f53b415a950 con 0x7f53b4156040 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f53b415a670 con 0x7f53b415b150 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4151fc0 con 0x7f53b415b930 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bc798640 10 monclient(hunting): authenticate will time out at 1134.111471s 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2503006318 0 0) 0x7f4230003620 con 0x7f4244155e60 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 >> v2:10.20.193.68:3300/0 conn(0x7f42440c9470 msgr2=0x7f42440d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 --2- 10.20.193.51:0/3362931827 >> v2:10.20.193.68:3300/0 conn(0x7f42440c9470 0x7f42440d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 >> v1:10.20.193.51:6789/0 conn(0x7f424415c790 legacy=0x7f42440c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42440cc730 con 0x7f4244155e60 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f4238002f00 con 0x7f4244155e60 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: dump: 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 >> v1:10.20.193.125:6789/0 conn(0x7f4244155e60 legacy=0x7f42440cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230003b50 0x7f4230003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): picked mon.a con 0x7f4230003b50 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 0x7f4230004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): picked mon.c con 0x7f4230004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4230005470 0x7f423000bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f53bad0e640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): picked mon.b con 0x7f4230005470 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f42440cc200 con 0x7f4230003b50 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 1 Processor -- start 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 1 -- start start 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f423000c3c0 con 0x7f4230005470 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f423000c5c0 con 0x7f4230004900 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f4241d74640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:38.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.531+0000 7f28f97b2640 10 monclient: init 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43760/0 (socket says 10.20.193.51:43760) 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 1 -- 10.20.193.51:0/3540113223 learned_addr learned my addr 10.20.193.51:0/3540113223 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f99101526a0 0x7f99100733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60916/0 (socket says 10.20.193.51:60916) 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 1 -- 10.20.193.51:0/1491815557 learned_addr learned my addr 10.20.193.51:0/1491815557 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 10 monclient(hunting): get_auth_request con 0x7f99101526a0 auth_method 0 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f990effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): get_auth_request con 0x7f53b415b930 auth_method 0 2026-02-22T21:37:38.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient: _add_conns ranks=[0,1,4,3,2,5] 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28f4156b30 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f28f4156f10 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f28f40743e0 0x7f28f4074830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-b con 0x7f28f40743e0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f4074d70 con 0x7f28f4156b30 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f4161ee0 con 0x7f28f4156f10 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f28f415e960 con 0x7f28f40743e0 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f28f97b2640 10 monclient(hunting): authenticate will time out at 1134.112403s 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f9e1a575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f9e1a575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f9e1a575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.532+0000 7f53bad0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.533+0000 7f4242575640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 0x7f4230004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 -- 10.20.193.51:0/3540113223 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b415b150 msgr2=0x7f53b40737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b415b150 0x7f53b40737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 -- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4156040 msgr2=0x7f53b41fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4156040 0x7f53b41fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f53bad0e640 1 -- 10.20.193.51:0/3540113223 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b40c6160 con 0x7f53b415b930 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f4242575640 10 monclient(hunting): get_auth_request con 0x7f4230004900 auth_method 0 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f4242575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f4242575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.534+0000 7f4242575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 10 monclient: monmap: 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.535836+0000 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.535836+0000 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2026-02-22T21:37:38.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 1 -- 10.20.193.51:0/1905418106 >> v2:10.20.193.68:3300/0 conn(0x7f9e1c0c8330 msgr2=0x7f9e1c1645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 1 --2- 10.20.193.51:0/1905418106 >> v2:10.20.193.68:3300/0 conn(0x7f9e1c0c8330 0x7f9e1c1645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 1 -- 10.20.193.51:0/1905418106 >> v1:10.20.193.51:6789/0 conn(0x7f9e1c154120 legacy=0x7f9e1c0c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f9e1a575640 1 -- 10.20.193.51:0/1905418106 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c15d4b0 con 0x7f9e1c153d40 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f4242575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f4242575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f4242575640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 1 Processor -- start 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 1 -- start start 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.535+0000 7f7d55232640 10 monclient: init 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f3e5db55640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f3e5db55640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f3e5db55640 10 monclient: monmap: 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.536855+0000 2026-02-22T21:37:38.556 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.536855+0000 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient: _add_conns ranks=[5,2,1,0,4,3] 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7d501526a0 0x7f7d500733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-c con 0x7f7d501526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d50152a80 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f7d5015d3b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 10 monclient(hunting): handle_auth_done global_id 4451 payload 293 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 -- 10.20.193.51:0/3362931827 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4230005470 msgr2=0x7f423000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4230005470 0x7f423000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d50073d60 con 0x7f7d5015d3b0 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 -- 10.20.193.51:0/3362931827 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230003b50 msgr2=0x7f4230003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230003b50 0x7f4230003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 monclient: found mon.c 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d50074040 con 0x7f7d50152a80 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4243577640 1 -- 10.20.193.51:0/3362931827 reap_dead start 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f7d500cca00 con 0x7f7d501526a0 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 1 -- 10.20.193.51:0/3362931827 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42440cc730 con 0x7f4230004900 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f4242575640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f7d55232640 10 monclient(hunting): authenticate will time out at 1134.116668s 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f42490c6640 5 monclient: authenticate success, global_id 4451 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f42490c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f53bad0e640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d250 tx=0x7f53b0004bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f539affd640 10 client.?.objecter ms_handle_connect 0x7f53b415b930 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f539affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f990effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f990effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.536+0000 7f990effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.537+0000 7f9e1a575640 1 --2- 10.20.193.51:0/1905418106 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c153d40 0x7f9e1c159ce0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c06e1d0 tx=0x7f9e10033410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d3e5c29bd6609f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 Processor -- start 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 -- start start 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient: init 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient: _add_conns ranks=[0,5,4,1,2,3] 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e5815f690 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f3e5815fa70 0x7f3e580cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-c con 0x7f3e5815fa70 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3e58155b50 0x7f3e58155f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-b con 0x7f3e58155b50 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e580ce0b0 con 0x7f3e5815f690 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f3e580ccb10 con 0x7f3e58155b50 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f3e580cdc20 con 0x7f3e5815fa70 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f3e5db55640 10 monclient(hunting): authenticate will time out at 1134.118743s 2026-02-22T21:37:38.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7f7d4ed76640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7d501526a0 0x7f7d500733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7fbe9321a640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7fbe9321a640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.538+0000 7fbe9321a640 10 monclient: monmap: 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.539540+0000 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.539540+0000 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 Processor -- start 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 -- start start 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient: init 2026-02-22T21:37:38.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f4242575640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 0x7f4230004d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f422c002f60 tx=0x7f422c02ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5265120789ae27e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f0eb29a9640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f0eb29a9640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f0eb29a9640 10 monclient: monmap: 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.540163+0000 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.540163+0000 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fb5de6f3640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fb5de6f3640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fb5de6f3640 10 monclient: monmap: 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.540414+0000 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.540414+0000 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.562 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7d501526a0 0x7f7d500733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43778/0 (socket says 10.20.193.51:43778) 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3864955986 learned_addr learned my addr 10.20.193.51:0/3864955986 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 10 monclient(hunting): get_auth_request con 0x7f7d501526a0 auth_method 0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7f7d4ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient: _add_conns ranks=[2,4,3,0,1,5] 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe8c155de0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbe8c1561c0 0x7fbe8c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe8c1561c0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe8c0caae0 0x7fbe8c0cc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8c0caae0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe8c0c9760 con 0x7fbe8c155de0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fbe8c0cc750 con 0x7fbe8c0caae0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fbe8c0c9a40 con 0x7fbe8c1561c0 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.539+0000 7fbe9321a640 10 monclient(hunting): authenticate will time out at 1134.119996s 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 Processor -- start 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 -- start start 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient: init 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 1 Processor -- start 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 1 -- start start 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient: init 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fedd52a0640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fedd52a0640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fedd52a0640 10 monclient: monmap: 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.540959+0000 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.540959+0000 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.564 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f587fdca640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f587fdca640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f587fdca640 10 monclient: monmap: 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.541228+0000 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.541228+0000 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 10 monclient(hunting): handle_auth_done global_id 4473 payload 274 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 1 -- 10.20.193.51:0/1491815557 >> v2:10.20.193.68:3300/0 conn(0x7f9910152a80 msgr2=0x7f99100cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 1 --2- 10.20.193.51:0/1491815557 >> v2:10.20.193.68:3300/0 conn(0x7f9910152a80 0x7f99100cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 1 -- 10.20.193.51:0/1491815557 >> v1:10.20.193.68:6789/0 conn(0x7f99100cdb20 legacy=0x7f991015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f990effd640 1 -- 10.20.193.51:0/1491815557 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9910074040 con 0x7f99101526a0 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient: _add_conns ranks=[0,4,1,5,3,2] 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5d81526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb5d8152a80 0x7fb5d80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d8152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fd23702b640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fd23702b640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fd23702b640 10 monclient: monmap: 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.541570+0000 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.541570+0000 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.566 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb5d80cdb20 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5d8073ae0 con 0x7fb5d81526a0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5d8073d60 con 0x7fb5d80cdb20 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fb5d80cca00 con 0x7fb5d8152a80 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fb5de6f3640 10 monclient(hunting): authenticate will time out at 1134.121065s 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 1 Processor -- start 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient: _add_conns ranks=[1,2,3,4,5,0] 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f0eac1526a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0eac152a80 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0eac15d3b0 0x7f0eac15d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0eac15d3b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac073ae0 con 0x7f0eac1526a0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac073d60 con 0x7f0eac152a80 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f0eac0cca00 con 0x7f0eac15d3b0 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.540+0000 7f0eb29a9640 10 monclient(hunting): authenticate will time out at 1134.120691s 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 Processor -- start 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 1 -- start start 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 10 monclient: init 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 -- start start 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient: init 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 1 Processor -- start 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 1 -- start start 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 10 monclient: init 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient: _add_conns ranks=[0,2,3,1,5,4] 2026-02-22T21:37:38.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedd0155e60 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedd015c790 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fedd00c9470 0x7fedd00d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-a con 0x7fedd00c9470 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd00cbc80 con 0x7fedd0155e60 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd00cc450 con 0x7fedd015c790 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fedd00cc200 con 0x7fedd00c9470 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fedd52a0640 10 monclient(hunting): authenticate will time out at 1134.121872s 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 10 monclient: _add_conns ranks=[2,3,4,1,0,5] 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2301526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7fd23702b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd230152a80 0x7fd2300cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 10 monclient: _add_conns ranks=[2,4,3,5,0,1] 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5878155c40 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.541+0000 7f587fdca640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f587815e760 0x7f5878068e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-a con 0x7fd230152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd2300cdb20 0x7fd23015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-b con 0x7fd2300cdb20 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd230073ae0 con 0x7fd2301526a0 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-b con 0x7f587815e760 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5878156020 0x7f587815f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-a con 0x7f5878156020 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fd2300cca00 con 0x7fd230152a80 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd230074c50 con 0x7fd2300cdb20 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7fd23702b640 10 monclient(hunting): authenticate will time out at 1134.122073s 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f587806ee10 con 0x7f5878155c40 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f58780693d0 con 0x7f5878156020 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f587815d320 con 0x7f587815e760 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f587fdca640 10 monclient(hunting): authenticate will time out at 1134.122114s 2026-02-22T21:37:38.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f990effd640 1 --2- 10.20.193.51:0/1491815557 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99100733d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f991015f410 tx=0x7f990402ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1492b2b1057427 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f3e577fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f3e5815f690 0x7f3e5815d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46974/0 (socket says 10.20.193.51:46974) 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f3e577fe640 1 -- 10.20.193.51:0/2192875529 learned_addr learned my addr 10.20.193.51:0/2192875529 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f6f23f40640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.542+0000 7f6f23f40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f6f23f40640 10 monclient: monmap: 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.543678+0000 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.543678+0000 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3864955986 >> v1:10.20.193.125:6789/0 conn(0x7f7d50152a80 legacy=0x7f7d500cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3864955986 >> v1:10.20.193.68:6789/0 conn(0x7f7d5015d3b0 legacy=0x7f7d5015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3864955986 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d50074320 con 0x7f7d501526a0 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b000b330 con 0x7f53b415b930 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f6f23f40640 1 Processor -- start 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: dump: 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f6f23f40640 1 -- start start 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f6f23f40640 10 monclient: init 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c002b30 con 0x7f4230004900 2026-02-22T21:37:38.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f4241d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f4241d74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f4241d74640 10 monclient: dump: 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f42490c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b000b4d0 con 0x7f53b415b930 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b00089b0 con 0x7f53b415b930 2026-02-22T21:37:38.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f539affd640 10 monclient: dump: 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 5 monclient: authenticate success, global_id 4454 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 10 monclient: _renew_subs 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b40c6440 con 0x7f53b415b930 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53b8d0a640 4 set_mon_vals no callback set 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 10 monclient: _renew_subs 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b41f9240 con 0x7f53b415b930 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f53bc798640 1 librados: init done 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.543+0000 7f28f2ffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f28f4156b30 0x7f28f415e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46962/0 (socket says 10.20.193.51:46962) 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f28f2ffd640 1 -- 10.20.193.51:0/986871263 learned_addr learned my addr 10.20.193.51:0/986871263 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f53bc798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f53bc798640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f53bc798640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b4152390 con 0x7f53b415b930 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient: _add_conns ranks=[4,1,0,3,2,5] 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f6f1c1526a0 0x7f6f1c0733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-b con 0x7f6f1c1526a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f6f1c152a80 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f1c15d3b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1c073d60 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1c074040 con 0x7f6f1c152a80 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c0cca00 con 0x7f6f1c1526a0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7f6f23f40640 10 monclient(hunting): authenticate will time out at 1134.124266s 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7fe24ad40640 10 monclient: get_monmap_and_config 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7fe24ad40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7fe24ad40640 10 monclient: monmap: 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:38.545033+0000 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:38.545033+0000 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.544+0000 7fe24ad40640 1 Processor -- start 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 1 -- start start 2026-02-22T21:37:38.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient: init 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f422c033070 con 0x7f4230004900 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 1 -- 10.20.193.51:0/3362931827 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c0054d0 con 0x7f4230004900 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 4 set_mon_vals no callback set 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f4241d74640 10 monclient: dump: 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient: _add_conns ranks=[2,1,5,0,3,4] 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe244155e60 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe24415c790 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe2440c9470 0x7fe2440d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-c con 0x7fe2440c9470 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2440cbc80 con 0x7fe24415c790 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2440cc450 con 0x7fe244155e60 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe2440cc200 con 0x7fe2440c9470 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7fe24ad40640 10 monclient(hunting): authenticate will time out at 1134.125555s 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 10 monclient: shutdown 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 1 -- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 msgr2=0x7f4230004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.545+0000 7f42490c6640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 0x7f4230004d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f422c002f60 tx=0x7f422c02ecb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 -- 10.20.193.51:0/3362931827 shutdown_connections 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 --2- 10.20.193.51:0/3362931827 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4230004900 0x7f4230004d80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 -- 10.20.193.51:0/3362931827 >> 10.20.193.51:0/3362931827 conn(0x7f424406d8c0 msgr2=0x7f424406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 -- 10.20.193.51:0/3362931827 shutdown_connections 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 -- 10.20.193.51:0/3362931827 wait complete. 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 10 monclient: monmap: 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f7d4ed76640 1 --2- 10.20.193.51:0/3864955986 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d501526a0 0x7f7d500733d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009970 tx=0x7f7d4402ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5dca123583e3470 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 librados: starting msgr at 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 librados: starting objecter 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.546+0000 7f42490c6640 1 Processor -- start 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 -- start start 2026-02-22T21:37:38.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 librados: setting wanted keys 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 librados: calling monclient init 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient: init 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 1 -- 10.20.193.51:0/1905418106 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1003f070 con 0x7f9e1c153d40 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: dump: 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 1 -- 10.20.193.51:0/1905418106 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e10038710 con 0x7f9e1c153d40 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 5 monclient: authenticate success, global_id 4470 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 1 -- 10.20.193.51:0/1905418106 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10038ae0 con 0x7f9e1c153d40 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e19573640 10 monclient: dump: 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.578 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 4 set_mon_vals no callback set 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4230004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f424406e450 0x7f42440d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-b con 0x7f424406e450 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4244150100 0x7f42440d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4244150100 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f42440ccfd0 con 0x7f4230004900 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f42440cccf0 con 0x7f424406e450 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f42440cd2b0 con 0x7f4244150100 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f42490c6640 10 monclient(hunting): authenticate will time out at 1134.127516s 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 10 monclient: shutdown 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 -- 10.20.193.51:0/1905418106 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c153d40 msgr2=0x7f9e1c159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 --2- 10.20.193.51:0/1905418106 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c153d40 0x7f9e1c159ce0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c06e1d0 tx=0x7f9e10033410 comp rx=0 tx=0).stop 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 -- 10.20.193.51:0/1905418106 shutdown_connections 2026-02-22T21:37:38.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 --2- 10.20.193.51:0/1905418106 >> v2:10.20.193.68:3300/0 conn(0x7f9e1c0c8330 0x7f9e1c1645f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 --2- 10.20.193.51:0/1905418106 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c153d40 0x7f9e1c159ce0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 -- 10.20.193.51:0/1905418106 >> 10.20.193.51:0/1905418106 conn(0x7f9e1c0cb010 msgr2=0x7f9e1c0cb420 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 -- 10.20.193.51:0/1905418106 shutdown_connections 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.547+0000 7f9e20e58640 1 -- 10.20.193.51:0/1905418106 wait complete. 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 10 monclient: monmap: 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 1 -- 10.20.193.51:0/1491815557 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990403c070 con 0x7f99101526a0 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 librados: starting msgr at 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 librados: starting objecter 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 Processor -- start 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7fbe91790640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe8c0caae0 0x7fbe8c0cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 -- start start 2026-02-22T21:37:38.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 librados: setting wanted keys 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: dump: 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 1 -- 10.20.193.51:0/1491815557 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990402fbd0 con 0x7f99101526a0 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 1 librados: calling monclient init 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 1 -- 10.20.193.51:0/1491815557 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904037440 con 0x7f99101526a0 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9915a68640 5 monclient: authenticate success, global_id 4473 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9915a68640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9915a68640 4 set_mon_vals no callback set 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f990dffb640 10 monclient: dump: 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.581 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.548+0000 7f9e20e58640 10 monclient: init 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e1c0c8330 0x7f9e1c1f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-c con 0x7f9e1c0c8330 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e1c153d40 0x7f9e1c0cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1c153d40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 10 monclient: shutdown 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 -- 10.20.193.51:0/1491815557 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 msgr2=0x7f99100733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 --2- 10.20.193.51:0/1491815557 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99100733d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f991015f410 tx=0x7f990402ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:38.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1c154120 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c15a220 con 0x7f9e1c154120 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c15da70 con 0x7f9e1c153d40 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c0c88f0 con 0x7f9e1c0c8330 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9e20e58640 10 monclient(hunting): authenticate will time out at 1134.129123s 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 -- 10.20.193.51:0/1491815557 shutdown_connections 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 --2- 10.20.193.51:0/1491815557 >> v2:10.20.193.68:3300/0 conn(0x7f9910152a80 0x7f99100cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 --2- 10.20.193.51:0/1491815557 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99100733d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765239904 0 0) 0x7f28f4074d70 con 0x7f28f4156b30 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 10 monclient(hunting): my global_id is 4479 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28e0003620 con 0x7f28f4156b30 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 -- 10.20.193.51:0/1491815557 >> 10.20.193.51:0/1491815557 conn(0x7f99100ce350 msgr2=0x7f99100ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 -- 10.20.193.51:0/1491815557 shutdown_connections 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 1 -- 10.20.193.51:0/1491815557 wait complete. 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 1 -- 10.20.193.51:0/2192875529 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975221611 0 0) 0x7f3e580ce0b0 con 0x7f3e5815f690 2026-02-22T21:37:38.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f0eb0f1f640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0eac15d3b0 0x7f0eac15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f9915a68640 10 monclient: monmap: 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 10 monclient(hunting): my global_id is 4476 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.549+0000 7f3e567fc640 1 -- 10.20.193.51:0/2192875529 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e3c003620 con 0x7f3e5815f690 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0eac15d3b0 0x7f0eac15d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60944/0 (socket says 10.20.193.51:60944) 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 1 -- 10.20.193.51:0/437952754 learned_addr learned my addr 10.20.193.51:0/437952754 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): get_auth_request con 0x7f0eac15d3b0 auth_method 0 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f0eb0f1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 librados: starting msgr at 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 librados: starting objecter 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 Processor -- start 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 -- start start 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 librados: setting wanted keys 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 librados: calling monclient init 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient: init 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f99100cdb20 0x7f99101f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-c con 0x7f99100cdb20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-a con 0x7f99101526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9910152a80 0x7f99101e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): picked mon.noname-b con 0x7f9910152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f99100748e0 con 0x7f99101526a0 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9910074600 con 0x7f9910152a80 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9910073910 con 0x7f99100cdb20 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.550+0000 7f9915a68640 10 monclient(hunting): authenticate will time out at 1134.130909s 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 10 monclient(hunting): handle_auth_done global_id 4482 payload 274 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 1 -- 10.20.193.51:0/437952754 >> v1:10.20.193.125:6789/0 conn(0x7f0eac152a80 legacy=0x7f0eac0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 1 -- 10.20.193.51:0/437952754 >> v1:10.20.193.68:6789/0 conn(0x7f0eac1526a0 legacy=0x7f0eac0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f0eb0f1f640 1 -- 10.20.193.51:0/437952754 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac074040 con 0x7f0eac15d3b0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5878156020 0x7f587815f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd230152a80 0x7fd2300cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5878156020 0x7f587815f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60978/0 (socket says 10.20.193.51:60978) 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 1 -- 10.20.193.51:0/1174597453 learned_addr learned my addr 10.20.193.51:0/1174597453 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 10 monclient(hunting): get_auth_request con 0x7f5878156020 auth_method 0 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7f587e340640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd230152a80 0x7fd2300cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60966/0 (socket says 10.20.193.51:60966) 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 1 -- 10.20.193.51:0/3311677647 learned_addr learned my addr 10.20.193.51:0/3311677647 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 10 monclient(hunting): get_auth_request con 0x7fd230152a80 auth_method 0 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fd227fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fbe91790640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe8c0caae0 0x7fbe8c0cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60936/0 (socket says 10.20.193.51:60936) 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fbe91790640 1 -- 10.20.193.51:0/851246684 learned_addr learned my addr 10.20.193.51:0/851246684 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.551+0000 7fbe91790640 10 monclient(hunting): get_auth_request con 0x7fbe8c0caae0 auth_method 0 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7f587e340640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7f587e340640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7f587e340640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fd227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fd227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fd227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.552+0000 7fbe91790640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 10 monclient(hunting): handle_auth_done global_id 4485 payload 274 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 1 -- 10.20.193.51:0/1174597453 >> v2:10.20.193.68:3300/0 conn(0x7f587815e760 msgr2=0x7f5878068e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 1 --2- 10.20.193.51:0/1174597453 >> v2:10.20.193.68:3300/0 conn(0x7f587815e760 0x7f5878068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 1 -- 10.20.193.51:0/1174597453 >> v1:10.20.193.125:6789/0 conn(0x7f5878155c40 legacy=0x7f587815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.553+0000 7f587e340640 1 -- 10.20.193.51:0/1174597453 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587806f0f0 con 0x7f5878156020 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 1 -- 10.20.193.51:0/3311677647 >> v2:10.20.193.68:3300/0 conn(0x7fd2300cdb20 msgr2=0x7fd23015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 1 --2- 10.20.193.51:0/3311677647 >> v2:10.20.193.68:3300/0 conn(0x7fd2300cdb20 0x7fd23015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 1 -- 10.20.193.51:0/3311677647 >> v1:10.20.193.125:6789/0 conn(0x7fd2301526a0 legacy=0x7fd2300733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.554+0000 7fd227fff640 1 -- 10.20.193.51:0/3311677647 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd230073d60 con 0x7fd230152a80 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 10 monclient(hunting): handle_auth_done global_id 4491 payload 274 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 1 -- 10.20.193.51:0/851246684 >> v2:10.20.193.68:3300/0 conn(0x7fbe8c1561c0 msgr2=0x7fbe8c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 1 --2- 10.20.193.51:0/851246684 >> v2:10.20.193.68:3300/0 conn(0x7fbe8c1561c0 0x7fbe8c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 1 -- 10.20.193.51:0/851246684 >> v1:10.20.193.125:6789/0 conn(0x7fbe8c155de0 legacy=0x7fbe8c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.555+0000 7fbe91790640 1 -- 10.20.193.51:0/851246684 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c0c9d20 con 0x7fbe8c0caae0 2026-02-22T21:37:38.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.556+0000 7f587e340640 1 --2- 10.20.193.51:0/1174597453 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5878156020 0x7f587815f420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f587815d4a0 tx=0x7f587402f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c063f6bbc42a9a88 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.557+0000 7fbe91790640 1 --2- 10.20.193.51:0/851246684 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c0cc210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0bb7e0 tx=0x7fbe8802f270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824a376ca3d2dc0d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.557+0000 7fb5d7fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fb5d81526a0 0x7fb5d80733d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:46990/0 (socket says 10.20.193.51:46990) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.557+0000 7fb5d7fff640 1 -- 10.20.193.51:0/1355321706 learned_addr learned my addr 10.20.193.51:0/1355321706 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.558+0000 7fd227fff640 1 --2- 10.20.193.51:0/3311677647 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd230152a80 0x7fd2300cd5e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd23008f0d0 tx=0x7fd21802ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25d96cbd33a6d0f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fedd00c9470 0x7fedd00d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7f9e1ad76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fedd00c9470 0x7fedd00d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60950/0 (socket says 10.20.193.51:60950) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 1 -- 10.20.193.51:0/2971236431 learned_addr learned my addr 10.20.193.51:0/2971236431 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 10 monclient(hunting): get_auth_request con 0x7fedd00c9470 auth_method 0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.560+0000 7fedcf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60990/0 (socket says 10.20.193.51:60990) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 1 -- 10.20.193.51:0/2838770376 learned_addr learned my addr 10.20.193.51:0/2838770376 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 10 monclient(hunting): get_auth_request con 0x7f9e1c154120 auth_method 0 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.561+0000 7f9e1ad76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7fedcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7fedcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7fedcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7f9e1ad76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7f9e1ad76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.562+0000 7f9e1ad76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 10 monclient(hunting): handle_auth_done global_id 4497 payload 274 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 1 -- 10.20.193.51:0/2971236431 >> v1:10.20.193.125:6789/0 conn(0x7fedd015c790 legacy=0x7fedd00c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 1 -- 10.20.193.51:0/2971236431 >> v1:10.20.193.51:6789/0 conn(0x7fedd0155e60 legacy=0x7fedd00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.563+0000 7fedcf7fe640 1 -- 10.20.193.51:0/2971236431 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd00cc730 con 0x7fedd00c9470 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 -- 10.20.193.51:0/2838770376 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e1c0c8330 msgr2=0x7f9e1c1f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e1c0c8330 0x7f9e1c1f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 -- 10.20.193.51:0/2838770376 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e1c153d40 msgr2=0x7f9e1c0cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e1c153d40 0x7f9e1c0cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.565+0000 7f9e1ad76640 1 -- 10.20.193.51:0/2838770376 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c0ce280 con 0x7f9e1c154120 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.567+0000 7f0eb0f1f640 1 --2- 10.20.193.51:0/437952754 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac15d3b0 0x7f0eac15d820 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea002f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6bb763b5fc423f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.567+0000 7f9e1ad76640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00ba10 tx=0x7f9e0c00bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.567+0000 7f9e02ffd640 10 client.?.objecter ms_handle_connect 0x7f9e1c154120 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.567+0000 7f9e02ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.567+0000 7fedcf7fe640 1 --2- 10.20.193.51:0/2971236431 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd00d1810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009e50 tx=0x7fedc002f2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d46096a0d120108 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.571+0000 7f6f224b6640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f6f1c15d3b0 0x7f6f1c15f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47016/0 (socket says 10.20.193.51:47016) 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.571+0000 7f6f224b6640 1 -- 10.20.193.51:0/1988505003 learned_addr learned my addr 10.20.193.51:0/1988505003 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.573+0000 7f4242d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 1 -- 10.20.193.51:0/1174597453 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587402fd50 con 0x7f5878156020 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: dump: 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.591 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 1 -- 10.20.193.51:0/1174597453 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58740373f0 con 0x7f5878156020 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 1 -- 10.20.193.51:0/1174597453 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58740377c0 con 0x7f5878156020 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 5 monclient: authenticate success, global_id 4485 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 4 set_mon_vals no callback set 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587cb3d640 10 monclient: dump: 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60980/0 (socket says 10.20.193.51:60980) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 1 -- 10.20.193.51:0/2992412763 learned_addr learned my addr 10.20.193.51:0/2992412763 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 10 monclient(hunting): get_auth_request con 0x7f4230004900 auth_method 0 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f4242d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 10 monclient: shutdown 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 -- 10.20.193.51:0/1174597453 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5878156020 msgr2=0x7f587815f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 --2- 10.20.193.51:0/1174597453 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5878156020 0x7f587815f420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f587815d4a0 tx=0x7f587402f2f0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 -- 10.20.193.51:0/1174597453 shutdown_connections 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 --2- 10.20.193.51:0/1174597453 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5878156020 0x7f587815f420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 1 -- 10.20.193.51:0/1355321706 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816029859 0 0) 0x7fb5c4002920 con 0x7fb5d81526a0 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 10 monclient(hunting): my global_id is 4494 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7fb5d6ffd640 1 -- 10.20.193.51:0/1355321706 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5bc003620 con 0x7fb5d81526a0 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 --2- 10.20.193.51:0/1174597453 >> v2:10.20.193.68:3300/0 conn(0x7f587815e760 0x7f5878068e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 -- 10.20.193.51:0/1174597453 >> 10.20.193.51:0/1174597453 conn(0x7f5878073f50 msgr2=0x7f5878074360 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 -- 10.20.193.51:0/1174597453 shutdown_connections 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.574+0000 7f587fdca640 1 -- 10.20.193.51:0/1174597453 wait complete. 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f4242d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f4242d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f4242d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 10 monclient: monmap: 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 librados: starting msgr at 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 librados: starting objecter 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 Processor -- start 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 -- start start 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 librados: setting wanted keys 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 1 librados: calling monclient init 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f587fdca640 10 monclient: init 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:60998/0 (socket says 10.20.193.51:60998) 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 1 -- 10.20.193.51:0/1229746078 learned_addr learned my addr 10.20.193.51:0/1229746078 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 10 monclient(hunting): get_auth_request con 0x7f99101526a0 auth_method 0 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.575+0000 7f990e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-b con 0x7f5878155c40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5878156020 0x7f5878159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-c con 0x7f5878156020 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f587815e760 0x7f5878151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): picked mon.noname-a con 0x7f587815e760 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f587815d850 con 0x7f587815e760 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f587815dbc0 con 0x7f5878155c40 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f587815f960 con 0x7f5878156020 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.576+0000 7f587fdca640 10 monclient(hunting): authenticate will time out at 1134.156246s 2026-02-22T21:37:38.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 -- 10.20.193.51:0/2992412763 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4244150100 msgr2=0x7f42440d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4244150100 0x7f42440d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 -- 10.20.193.51:0/2992412763 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f424406e450 msgr2=0x7f42440d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f424406e450 0x7f42440d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.577+0000 7f4242d76640 1 -- 10.20.193.51:0/2992412763 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4244152760 con 0x7f4230004900 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.578+0000 7f990e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.578+0000 7f990e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.578+0000 7f990e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.579+0000 7f42237fe640 10 client.?.objecter ms_handle_connect 0x7f4230004900 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.579+0000 7f42237fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.579+0000 7f4242d76640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f423800c860 tx=0x7f423800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 -- 10.20.193.51:0/1229746078 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f99100cdb20 msgr2=0x7f99101f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f99100cdb20 0x7f99101f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 -- 10.20.193.51:0/1229746078 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9910152a80 msgr2=0x7f99101e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9910152a80 0x7f99101e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.580+0000 7f990e7fc640 1 -- 10.20.193.51:0/1229746078 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99101f4940 con 0x7f99101526a0 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 1 -- 10.20.193.51:0/3311677647 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21803c070 con 0x7fd230152a80 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: dump: 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 1 -- 10.20.193.51:0/3311677647 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21802fbd0 con 0x7fd230152a80 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 1 -- 10.20.193.51:0/3311677647 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218037440 con 0x7fd230152a80 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd23702b640 5 monclient: authenticate success, global_id 4488 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd23702b640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd23702b640 4 set_mon_vals no callback set 2026-02-22T21:37:38.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.583+0000 7fd2277fe640 10 monclient: dump: 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 10 monclient: shutdown 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- 10.20.193.51:0/3311677647 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd230152a80 msgr2=0x7fd2300cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 --2- 10.20.193.51:0/3311677647 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd230152a80 0x7fd2300cd5e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd23008f0d0 tx=0x7fd21802ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- 10.20.193.51:0/3311677647 shutdown_connections 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 --2- 10.20.193.51:0/3311677647 >> v2:10.20.193.68:3300/0 conn(0x7fd2300cdb20 0x7fd23015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 --2- 10.20.193.51:0/3311677647 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd230152a80 0x7fd2300cd5e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- 10.20.193.51:0/3311677647 >> 10.20.193.51:0/3311677647 conn(0x7fd2300ce350 msgr2=0x7fd2300ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- 10.20.193.51:0/3311677647 shutdown_connections 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- 10.20.193.51:0/3311677647 wait complete. 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 10 monclient: monmap: 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 librados: starting msgr at 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 librados: starting objecter 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 Processor -- start 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 -- start start 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 librados: setting wanted keys 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 1 librados: calling monclient init 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.584+0000 7fd23702b640 10 monclient: init 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-b con 0x7fd2300cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd2301526a0 0x7fd23014ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-a con 0x7fd2301526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd230152a80 0x7fd230150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): picked mon.noname-c con 0x7fd230152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd230074600 con 0x7fd2301526a0 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd230074320 con 0x7fd2300cdb20 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd2300748e0 con 0x7fd230152a80 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fd23702b640 10 monclient(hunting): authenticate will time out at 1134.165549s 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c00c780 con 0x7f9e1c154120 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: dump: 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c010070 con 0x7f9e1c154120 2026-02-22T21:37:38.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 5 monclient: authenticate success, global_id 4500 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 10 monclient: _renew_subs 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c159060 con 0x7f9e1c154120 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c00cb80 con 0x7f9e1c154120 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e18d72640 4 set_mon_vals no callback set 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e02ffd640 10 monclient: dump: 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 10 monclient: _renew_subs 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c159620 con 0x7f9e1c154120 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f9e20e58640 1 librados: init done 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f990e7fc640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b780 tx=0x7f98f800bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f98e7fff640 10 client.?.objecter ms_handle_connect 0x7f99101526a0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7f98e7fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fbe837fe640 1 -- 10.20.193.51:0/851246684 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8802fcd0 con 0x7fbe8c0caae0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fbe837fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fbe837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.585+0000 7fbe837fe640 10 monclient: dump: 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.600 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 1 -- 10.20.193.51:0/851246684 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe880373f0 con 0x7fbe8c0caae0 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 5 monclient: authenticate success, global_id 4491 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 1 -- 10.20.193.51:0/851246684 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880377c0 con 0x7fbe8c0caae0 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe837fe640 10 monclient: dump: 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 4 set_mon_vals no callback set 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 1 -- 10.20.193.51:0/1988505003 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076061300 0 0) 0x7f6f1c073d60 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 10 monclient(hunting): _negotiate 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 10 monclient(hunting): my global_id is 4506 2026-02-22T21:37:38.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f6f20cb3640 1 -- 10.20.193.51:0/1988505003 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f04003620 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f9e20e58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f9e20e58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f9e20e58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c159fc0 con 0x7f9e1c154120 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 10 monclient: shutdown 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 -- 10.20.193.51:0/851246684 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 msgr2=0x7fbe8c0cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 --2- 10.20.193.51:0/851246684 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c0cc210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0bb7e0 tx=0x7fbe8802f270 comp rx=0 tx=0).stop 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 -- 10.20.193.51:0/851246684 shutdown_connections 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 --2- 10.20.193.51:0/851246684 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c0cc210 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 --2- 10.20.193.51:0/851246684 >> v2:10.20.193.68:3300/0 conn(0x7fbe8c1561c0 0x7fbe8c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 -- 10.20.193.51:0/851246684 >> 10.20.193.51:0/851246684 conn(0x7fbe8c0d1ec0 msgr2=0x7fbe8c0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 -- 10.20.193.51:0/851246684 shutdown_connections 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fbe9321a640 1 -- 10.20.193.51:0/851246684 wait complete. 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 1 -- 10.20.193.51:0/2971236431 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc002fd00 con 0x7fedd00c9470 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: dump: 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.602 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 1 -- 10.20.193.51:0/2971236431 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc00373f0 con 0x7fedd00c9470 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 1 -- 10.20.193.51:0/2971236431 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc00377c0 con 0x7fedd00c9470 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedd52a0640 5 monclient: authenticate success, global_id 4497 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedd52a0640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedd52a0640 4 set_mon_vals no callback set 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7fedcdffb640 10 monclient: dump: 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 201066655 0 0) 0x7f28e0003620 con 0x7f28f4156b30 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 >> v2:10.20.193.68:3300/0 conn(0x7f28f40743e0 msgr2=0x7f28f4074830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 --2- 10.20.193.51:0/986871263 >> v2:10.20.193.68:3300/0 conn(0x7f28f40743e0 0x7f28f4074830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 >> v1:10.20.193.68:6789/0 conn(0x7f28f4156f10 legacy=0x7f28f41617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:37:38.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f41621a0 con 0x7f28f4156b30 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.586+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f28e8002f00 con 0x7f28f4156b30 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient: dump: 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient: monmap: 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.604 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 >> v1:10.20.193.51:6789/0 conn(0x7f28f4156b30 legacy=0x7f28f415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28e0003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): picked mon.b con 0x7f28e0003ba0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28e0004900 0x7f28e0004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): picked mon.a con 0x7f28e0004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28e0005510 0x7f28e000bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): picked mon.c con 0x7f28e0005510 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f28f415e960 con 0x7f28e0004900 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f28e000c450 con 0x7f28e0003ba0 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f28e000c650 con 0x7f28e0005510 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7f28f1ffb640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 librados: starting msgr at 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 librados: starting objecter 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 10 monclient: shutdown 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 -- 10.20.193.51:0/2971236431 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 msgr2=0x7fedd00d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 --2- 10.20.193.51:0/2971236431 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd00d1810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009e50 tx=0x7fedc002f2a0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 Processor -- start 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 -- 10.20.193.51:0/2971236431 shutdown_connections 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 --2- 10.20.193.51:0/2971236431 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd00d1810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 -- 10.20.193.51:0/2971236431 >> 10.20.193.51:0/2971236431 conn(0x7fedd006d8c0 msgr2=0x7fedd006dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 -- start start 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 librados: setting wanted keys 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 librados: calling monclient init 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient: init 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 -- 10.20.193.51:0/2971236431 shutdown_connections 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 1 -- 10.20.193.51:0/2971236431 wait complete. 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c1eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8c0caae0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe8c155de0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fedd52a0640 10 monclient: monmap: 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe8c1561c0 0x7fbe8c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe8c1561c0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c0ca5c0 con 0x7fbe8c0caae0 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c0ca2e0 con 0x7fbe8c155de0 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.587+0000 7fbe9321a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c074fb0 con 0x7fbe8c1561c0 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fbe9321a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fbe9321a640 10 monclient(hunting): authenticate will time out at 1134.168009s 2026-02-22T21:37:38.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 librados: starting msgr at 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 librados: starting objecter 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 Processor -- start 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 -- start start 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 librados: setting wanted keys 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 librados: calling monclient init 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient: init 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd0151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-a con 0x7fedd00c9470 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-b con 0x7fedd0155e60 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fedd015c790 0x7fedd0147520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): picked mon.noname-c con 0x7fedd015c790 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fedd00ccfd0 con 0x7fedd00c9470 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fedd00cccf0 con 0x7fedd0155e60 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fedd00cd2b0 con 0x7fedd015c790 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.588+0000 7fedd52a0640 10 monclient(hunting): authenticate will time out at 1134.168954s 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 1 -- 10.20.193.51:0/437952754 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea002fd90 con 0x7f0eac15d3b0 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: dump: 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 1 -- 10.20.193.51:0/437952754 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ea00373f0 con 0x7f0eac15d3b0 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 5 monclient: authenticate success, global_id 4482 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 1 -- 10.20.193.51:0/437952754 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00377c0 con 0x7f0eac15d3b0 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 4 set_mon_vals no callback set 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eaaffd640 10 monclient: dump: 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 10 monclient: shutdown 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 -- 10.20.193.51:0/437952754 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac15d3b0 msgr2=0x7f0eac15d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/437952754 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac15d3b0 0x7f0eac15d820 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea002f330 comp rx=0 tx=0).stop 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 -- 10.20.193.51:0/437952754 shutdown_connections 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/437952754 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac15d3b0 0x7f0eac15d820 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 -- 10.20.193.51:0/437952754 >> 10.20.193.51:0/437952754 conn(0x7f0eac0ce350 msgr2=0x7f0eac0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 -- 10.20.193.51:0/437952754 shutdown_connections 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.589+0000 7f0eb29a9640 1 -- 10.20.193.51:0/437952754 wait complete. 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient: monmap: 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 librados: starting msgr at 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 librados: starting objecter 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 Processor -- start 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 -- start start 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 librados: setting wanted keys 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 librados: calling monclient init 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient: init 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac1526a0 0x7f0eac159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0eac1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-b con 0x7f0eac152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0eac15d3b0 0x7f0eac1571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0eac15d3b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac074600 con 0x7f0eac1526a0 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac074c50 con 0x7f0eac152a80 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac0748e0 con 0x7f0eac15d3b0 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.590+0000 7f0eb29a9640 10 monclient(hunting): authenticate will time out at 1134.170925s 2026-02-22T21:37:38.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.630+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f8a6800c040 con 0x7f8a7c156040 2026-02-22T21:37:38.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.631+0000 7f8a57fff640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 0x7f8a4c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.631+0000 7f8a7affd640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 0x7f8a4c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.631+0000 7f8a7affd640 10 monclient: get_auth_request con 0x7f8a4c078e80 auth_method 0 2026-02-22T21:37:38.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.631+0000 7f8a7affd640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 0x7f8a4c07b340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c159cd0 tx=0x7f8a6c034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.631+0000 7f8a57fff640 10 client.4474.objecter ms_handle_connect 0x7f8a4c078e80 2026-02-22T21:37:38.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.633+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f8a68004610 con 0x7f8a7c156040 2026-02-22T21:37:38.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.633+0000 7f8a57fff640 10 client.4474.objecter ms_dispatch2 0x7f8a7c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.633+0000 7f8a57fff640 3 client.4474.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.633+0000 7f8a57fff640 3 client.4474.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.633+0000 7f8a57fff640 20 client.4474.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.639+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f8a68063bc0 con 0x7f8a7c156040 2026-02-22T21:37:38.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.639+0000 7f8a57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.639+0000 7f8a57fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.642+0000 7fb5d77fe640 1 --2- 10.20.193.51:0/1355321706 >> v2:10.20.193.68:3300/0 conn(0x7fb5d8152a80 0x7fb5d80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.642+0000 7fb5d77fe640 10 monclient(hunting): get_auth_request con 0x7fb5d8152a80 auth_method 0 2026-02-22T21:37:38.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.642+0000 7fb5d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.642+0000 7fb5d77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.642+0000 7fb5d77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.643+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.643+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.643+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.644+0000 7f6f21cb5640 1 --2- 10.20.193.51:0/1988505003 >> v2:10.20.193.68:3300/0 conn(0x7f6f1c1526a0 0x7f6f1c0733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.644+0000 7f6f21cb5640 10 monclient(hunting): get_auth_request con 0x7f6f1c1526a0 auth_method 0 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.644+0000 7f6f21cb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.644+0000 7f6f21cb5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.644+0000 7f6f21cb5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_done global_id 4477 payload 274 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 1 -- 10.20.193.51:0/1355321706 >> v1:10.20.193.68:6789/0 conn(0x7fb5d80cdb20 legacy=0x7fb5d815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 1 -- 10.20.193.51:0/1355321706 >> v1:10.20.193.51:6789/0 conn(0x7fb5d81526a0 legacy=0x7fb5d80733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7fb5d77fe640 1 -- 10.20.193.51:0/1355321706 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d8074040 con 0x7fb5d8152a80 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7f6f21cb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7f6f21cb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.645+0000 7f6f21cb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 10 monclient(hunting): handle_auth_done global_id 4480 payload 274 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 1 -- 10.20.193.51:0/1988505003 >> v1:10.20.193.68:6789/0 conn(0x7f6f1c152a80 legacy=0x7f6f1c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2192875529 >> v2:10.20.193.125:3300/0 conn(0x7f3e5815fa70 0x7f3e580cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 1 -- 10.20.193.51:0/1988505003 >> v1:10.20.193.51:6789/0 conn(0x7f6f1c15d3b0 legacy=0x7f6f1c15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7f6f21cb5640 1 -- 10.20.193.51:0/1988505003 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c074320 con 0x7f6f1c1526a0 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fb5d77fe640 1 --2- 10.20.193.51:0/1355321706 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d8152a80 0x7fb5d80cd5e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80098a0 tx=0x7fb5c802eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5584735bff52cae9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe2440c9470 0x7fe2440d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe2440c9470 0x7fe2440d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43794/0 (socket says 10.20.193.51:43794) 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 1 -- 10.20.193.51:0/4162109198 learned_addr learned my addr 10.20.193.51:0/4162109198 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 10 monclient(hunting): get_auth_request con 0x7fe2440c9470 auth_method 0 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.647+0000 7fe2492b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): get_auth_request con 0x7f3e5815fa70 auth_method 0 2026-02-22T21:37:38.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.648+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 1 -- 10.20.193.51:0/4162109198 >> v1:10.20.193.125:6789/0 conn(0x7fe244155e60 legacy=0x7fe2440cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 1 -- 10.20.193.51:0/4162109198 >> v1:10.20.193.68:6789/0 conn(0x7fe24415c790 legacy=0x7fe2440c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7fe2492b6640 1 -- 10.20.193.51:0/4162109198 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2440cc730 con 0x7fe2440c9470 2026-02-22T21:37:38.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f6f21cb5640 1 --2- 10.20.193.51:0/1988505003 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c0733d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0098a0 tx=0x7f6f0c02eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a41c1373ce146f69 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_done global_id 4466 payload 274 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2192875529 >> v2:10.20.193.68:3300/0 conn(0x7f3e58155b50 msgr2=0x7f3e58155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2192875529 >> v2:10.20.193.68:3300/0 conn(0x7f3e58155b50 0x7f3e58155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2192875529 >> v1:10.20.193.51:6789/0 conn(0x7f3e5815f690 legacy=0x7f3e5815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.650+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2192875529 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e580ccd30 con 0x7f3e5815fa70 2026-02-22T21:37:38.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.652+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2192875529 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815fa70 0x7f3e580cc5d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e580c5450 tx=0x7f3e4802ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15090c6f35fafdf6 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.652+0000 7fe2492b6640 1 --2- 10.20.193.51:0/4162109198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe2440c9470 0x7fe2440d1810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe240009e50 tx=0x7fe24002f2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898a36de7c0a1071 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35596/0 (socket says 10.20.193.51:35596) 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 1 -- 10.20.193.51:0/3413319099 learned_addr learned my addr 10.20.193.51:0/3413319099 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 10 monclient(hunting): get_auth_request con 0x7f5878155c40 auth_method 0 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.653+0000 7f587db3f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.654+0000 7fd234da0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.654+0000 7f587db3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.654+0000 7f587db3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.654+0000 7f587db3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35602/0 (socket says 10.20.193.51:35602) 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 1 -- 10.20.193.51:0/334719626 learned_addr learned my addr 10.20.193.51:0/334719626 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 10 monclient(hunting): get_auth_request con 0x7fd2300cdb20 auth_method 0 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.655+0000 7fd234da0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 -- 10.20.193.51:0/3413319099 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5878156020 msgr2=0x7f5878159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5878156020 0x7f5878159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 -- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f587815e760 msgr2=0x7f5878151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f587815e760 0x7f5878151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7f587db3f640 1 -- 10.20.193.51:0/3413319099 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58781e8420 con 0x7f5878155c40 2026-02-22T21:37:38.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.656+0000 7fbe83fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.657+0000 7fd234da0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.657+0000 7fd234da0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.657+0000 7fd234da0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35608/0 (socket says 10.20.193.51:35608) 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 1 -- 10.20.193.51:0/4282826210 learned_addr learned my addr 10.20.193.51:0/4282826210 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 10 monclient(hunting): get_auth_request con 0x7fbe8c155de0 auth_method 0 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7fbe83fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7f587db3f640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f586800b5a0 tx=0x7f586800ba70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7f58667fc640 10 client.?.objecter ms_handle_connect 0x7f5878155c40 2026-02-22T21:37:38.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.658+0000 7f58667fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 10 monclient(hunting): handle_auth_done global_id 4486 payload 824 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 -- 10.20.193.51:0/334719626 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd230152a80 msgr2=0x7fd230150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd230152a80 0x7fd230150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 -- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd2301526a0 msgr2=0x7fd23014ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd2301526a0 0x7fd23014ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fd234da0640 1 -- 10.20.193.51:0/334719626 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23014abb0 con 0x7fd2300cdb20 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fbe83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fbe83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.660+0000 7fbe83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.661+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28e0003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.661+0000 7f28f2ffd640 10 monclient(hunting): get_auth_request con 0x7f28e0003ba0 auth_method 0 2026-02-22T21:37:38.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.661+0000 7f28f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.661+0000 7f28f2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.661+0000 7f28f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:38.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.662+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f53b0044070 con 0x7f53b415b930 2026-02-22T21:37:38.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7f539affd640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7f53ba50d640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7f53ba50d640 10 monclient: get_auth_request con 0x7f5384078e50 auth_method 0 2026-02-22T21:37:38.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7f53ba50d640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009750 tx=0x7f53a4006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7f539affd640 10 client.4454.objecter ms_handle_connect 0x7f5384078e50 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 10 monclient(hunting): handle_auth_done global_id 4489 payload 824 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 -- 10.20.193.51:0/4282826210 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe8c1561c0 msgr2=0x7fbe8c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe8c1561c0 0x7fbe8c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 -- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 msgr2=0x7fbe8c1eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c1eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.663+0000 7fbe83fff640 1 -- 10.20.193.51:0/4282826210 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c14a2b0 con 0x7fbe8c155de0 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7f28f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7f28f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7f28f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7fd234da0640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b500 tx=0x7fd22000b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7fd2257fa640 10 client.?.objecter ms_handle_connect 0x7fd2300cdb20 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.664+0000 7fd2257fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.665+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f53b003b5b0 con 0x7f53b415b930 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.665+0000 7f539affd640 10 client.4454.objecter ms_dispatch2 0x7f53b4000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.665+0000 7f539affd640 3 client.4454.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.665+0000 7f539affd640 3 client.4454.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.665+0000 7f539affd640 20 client.4454.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 10 monclient(hunting): handle_auth_done global_id 4479 payload 293 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 -- 10.20.193.51:0/986871263 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28e0005510 msgr2=0x7f28e000bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28e0005510 0x7f28e000bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 -- 10.20.193.51:0/986871263 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28e0004900 msgr2=0x7f28e0004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28e0004900 0x7f28e0004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 monclient: found mon.b 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 1 -- 10.20.193.51:0/986871263 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e0003620 con 0x7f28e0003ba0 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f37fe640 1 -- 10.20.193.51:0/986871263 reap_dead start 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f97b2640 5 monclient: authenticate success, global_id 4479 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7f28f97b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7fbe83fff640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400b780 tx=0x7fbe7400bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7fbe817fa640 10 client.?.objecter ms_handle_connect 0x7fbe8c155de0 2026-02-22T21:37:38.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.666+0000 7fbe817fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.668+0000 7fedce7fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28e0003fa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c320 tx=0x7f28e802f5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1861eda9a8ed0bce server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35610/0 (socket says 10.20.193.51:35610) 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 1 -- 10.20.193.51:0/445221448 learned_addr learned my addr 10.20.193.51:0/445221448 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 10 monclient(hunting): get_auth_request con 0x7fedd0155e60 auth_method 0 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.669+0000 7fedce7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.670+0000 7fedce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.670+0000 7fedce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.670+0000 7fedce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.671+0000 7f0eab7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.671+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f53b0053020 con 0x7f53b415b930 2026-02-22T21:37:38.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.671+0000 7f539affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.671+0000 7f539affd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 10 monclient(hunting): handle_auth_done global_id 4492 payload 824 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 -- 10.20.193.51:0/445221448 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fedd015c790 msgr2=0x7fedd0147520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fedd015c790 0x7fedd0147520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 -- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 msgr2=0x7fedd0151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd0151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.672+0000 7fedce7fc640 1 -- 10.20.193.51:0/445221448 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd0150150 con 0x7fedd0155e60 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35616/0 (socket says 10.20.193.51:35616) 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 1 -- 10.20.193.51:0/4277624715 learned_addr learned my addr 10.20.193.51:0/4277624715 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 10 monclient(hunting): get_auth_request con 0x7f0eac152a80 auth_method 0 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.673+0000 7f0eab7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.674+0000 7f0eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.674+0000 7f0eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.674+0000 7f0eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.675+0000 7f0eb0f1f640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0eac15d3b0 0x7f0eac1571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eb0f1f640 10 monclient(hunting): get_auth_request con 0x7f0eac15d3b0 auth_method 0 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eb0f1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eb0f1f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eb0f1f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2026-02-22T21:37:38.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 -- 10.20.193.51:0/4277624715 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0eac15d3b0 msgr2=0x7f0eac1571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0eac15d3b0 0x7f0eac1571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 -- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac1526a0 msgr2=0x7f0eac159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac1526a0 0x7f0eac159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7f0eab7fe640 1 -- 10.20.193.51:0/4277624715 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1597d0 con 0x7f0eac152a80 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 1 -- 10.20.193.51:0/1355321706 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c803c070 con 0x7fb5d8152a80 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: dump: 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.679 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 1 -- 10.20.193.51:0/1355321706 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5c802fb70 con 0x7fb5d8152a80 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 1 -- 10.20.193.51:0/1355321706 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8037440 con 0x7fb5d8152a80 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5de6f3640 5 monclient: authenticate success, global_id 4477 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5de6f3640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5de6f3640 4 set_mon_vals no callback set 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.676+0000 7fb5d6ffd640 10 monclient: dump: 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 10 monclient: shutdown 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 -- 10.20.193.51:0/1355321706 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d8152a80 msgr2=0x7fb5d80cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/1355321706 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d8152a80 0x7fb5d80cd5e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80098a0 tx=0x7fb5c802eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 -- 10.20.193.51:0/1355321706 shutdown_connections 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/1355321706 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d8152a80 0x7fb5d80cd5e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 -- 10.20.193.51:0/1355321706 >> 10.20.193.51:0/1355321706 conn(0x7fb5d80ce350 msgr2=0x7fb5d80ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 -- 10.20.193.51:0/1355321706 shutdown_connections 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 -- 10.20.193.51:0/1355321706 wait complete. 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 10 monclient: monmap: 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 librados: starting msgr at 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.677+0000 7fb5de6f3640 1 librados: starting objecter 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 Processor -- start 2026-02-22T21:37:38.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 -- start start 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 librados: setting wanted keys 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 librados: calling monclient init 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient: init 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f0eab7fe640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00b4d0 tx=0x7f0e9c00b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f0ea8ff9640 10 client.?.objecter ms_handle_connect 0x7f0eac152a80 2026-02-22T21:37:38.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f0ea8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 1 -- 10.20.193.51:0/2192875529 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4803c070 con 0x7f3e5815fa70 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: dump: 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 1 -- 10.20.193.51:0/2192875529 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4802faf0 con 0x7f3e5815fa70 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 1 -- 10.20.193.51:0/2192875529 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48037440 con 0x7f3e5815fa70 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 5 monclient: authenticate success, global_id 4466 2026-02-22T21:37:38.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 4 set_mon_vals no callback set 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e567fc640 10 monclient: dump: 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d80cdb20 0x7fb5d814cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d80cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-c con 0x7fb5d81526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5d8152a80 0x7fb5d81f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): picked mon.noname-a con 0x7fb5d8152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8074600 con 0x7fb5d8152a80 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8074c50 con 0x7fb5d80cdb20 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d80748e0 con 0x7fb5d81526a0 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fb5de6f3640 10 monclient(hunting): authenticate will time out at 1134.258558s 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fedce7fc640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b500 tx=0x7fedb800b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fedaf7fe640 10 client.?.objecter ms_handle_connect 0x7fedd0155e60 2026-02-22T21:37:38.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fedaf7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 1 -- 10.20.193.51:0/4162109198 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24002fd00 con 0x7fe2440c9470 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: dump: 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 1 -- 10.20.193.51:0/4162109198 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2400373f0 con 0x7fe2440c9470 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 1 -- 10.20.193.51:0/4162109198 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2400377c0 con 0x7fe2440c9470 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe23b7fe640 10 monclient: dump: 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe24ad40640 5 monclient: authenticate success, global_id 4463 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe24ad40640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7fe24ad40640 4 set_mon_vals no callback set 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 10 monclient: shutdown 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 -- 10.20.193.51:0/2192875529 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815fa70 msgr2=0x7f3e580cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2192875529 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815fa70 0x7f3e580cc5d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e580c5450 tx=0x7f3e4802ede0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 -- 10.20.193.51:0/2192875529 shutdown_connections 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2192875529 >> v2:10.20.193.68:3300/0 conn(0x7f3e58155b50 0x7f3e58155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2192875529 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815fa70 0x7f3e580cc5d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 -- 10.20.193.51:0/2192875529 >> 10.20.193.51:0/2192875529 conn(0x7f3e580d1b50 msgr2=0x7f3e580d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 -- 10.20.193.51:0/2192875529 shutdown_connections 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.678+0000 7f3e5db55640 1 -- 10.20.193.51:0/2192875529 wait complete. 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 10 monclient: shutdown 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 -- 10.20.193.51:0/4162109198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe2440c9470 msgr2=0x7fe2440d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 --2- 10.20.193.51:0/4162109198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe2440c9470 0x7fe2440d1810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe240009e50 tx=0x7fe24002f2a0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 -- 10.20.193.51:0/4162109198 shutdown_connections 2026-02-22T21:37:38.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 --2- 10.20.193.51:0/4162109198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe2440c9470 0x7fe2440d1810 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 -- 10.20.193.51:0/4162109198 >> 10.20.193.51:0/4162109198 conn(0x7fe24406d8c0 msgr2=0x7fe24406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 -- 10.20.193.51:0/4162109198 shutdown_connections 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 1 -- 10.20.193.51:0/4162109198 wait complete. 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 10 monclient: monmap: 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 librados: starting msgr at 2026-02-22T21:37:38.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 librados: starting objecter 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 Processor -- start 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 -- start start 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 librados: setting wanted keys 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 1 librados: calling monclient init 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7f3e5db55640 10 monclient: init 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74004480 con 0x7fbe8c155de0 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: dump: 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.690 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe74010070 con 0x7fbe8c155de0 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74002920 con 0x7fbe8c155de0 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 5 monclient: authenticate success, global_id 4489 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe82ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe817fa640 10 monclient: dump: 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 10 monclient: _renew_subs 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8c0d0f60 con 0x7fbe8c155de0 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fe24ad40640 10 monclient: monmap: 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 10 monclient: _renew_subs 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8c14a520 con 0x7fbe8c155de0 2026-02-22T21:37:38.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.679+0000 7fbe9321a640 1 librados: init done 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 librados: starting msgr at 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 librados: starting objecter 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 Processor -- start 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 -- start start 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 librados: setting wanted keys 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 librados: calling monclient init 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient: init 2026-02-22T21:37:38.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3e58155b50 0x7f3e580c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-b con 0x7f3e58155b50 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-c con 0x7f3e5815f690 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3e5815fa70 0x7f3e581e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): picked mon.noname-a con 0x7f3e5815fa70 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f3e580cd5d0 con 0x7f3e5815fa70 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f3e580cd2f0 con 0x7f3e58155b50 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f3e580cd8b0 con 0x7f3e5815f690 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7f3e5db55640 10 monclient(hunting): authenticate will time out at 1134.260247s 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe2440c9470 0x7fe2441499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-a con 0x7fe2440c9470 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe244155e60 0x7fe24413eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-b con 0x7fe244155e60 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fbe9321a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): picked mon.noname-c con 0x7fe24415c790 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440ccfd0 con 0x7fe2440c9470 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fbe9321a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fbe9321a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440cccf0 con 0x7fe244155e60 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe2440cd2b0 con 0x7fe24415c790 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fe24ad40640 10 monclient(hunting): authenticate will time out at 1134.260593s 2026-02-22T21:37:38.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.680+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8c0cbee0 con 0x7fbe8c155de0 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.682+0000 7fb5d77fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43892/0 (socket says 10.20.193.51:43892) 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 1 -- 10.20.193.51:0/3831108424 learned_addr learned my addr 10.20.193.51:0/3831108424 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): get_auth_request con 0x7fb5d81526a0 auth_method 0 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.683+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 -- 10.20.193.51:0/3831108424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d80cdb20 msgr2=0x7fb5d814cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d80cdb20 0x7fb5d814cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 -- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5d8152a80 msgr2=0x7fb5d81f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5d8152a80 0x7fb5d81f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.684+0000 7fb5d77fe640 1 -- 10.20.193.51:0/3831108424 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d8157640 con 0x7fb5d81526a0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 1 -- 10.20.193.51:0/3864955986 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4403c070 con 0x7f7d501526a0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: dump: 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 1 -- 10.20.193.51:0/3864955986 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d4402fc30 con 0x7f7d501526a0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 1 -- 10.20.193.51:0/3864955986 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44037440 con 0x7f7d501526a0 2026-02-22T21:37:38.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 5 monclient: authenticate success, global_id 4457 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 4 set_mon_vals no callback set 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d4dd74640 10 monclient: dump: 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 10 monclient: shutdown 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 -- 10.20.193.51:0/3864955986 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d501526a0 msgr2=0x7f7d500733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 --2- 10.20.193.51:0/3864955986 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d501526a0 0x7f7d500733d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009970 tx=0x7f7d4402ef70 comp rx=0 tx=0).stop 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 -- 10.20.193.51:0/3864955986 shutdown_connections 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 --2- 10.20.193.51:0/3864955986 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d501526a0 0x7f7d500733d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 -- 10.20.193.51:0/3864955986 >> 10.20.193.51:0/3864955986 conn(0x7f7d500ce350 msgr2=0x7f7d500ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 -- 10.20.193.51:0/3864955986 shutdown_connections 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f7d55232640 1 -- 10.20.193.51:0/3864955986 wait complete. 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43902/0 (socket says 10.20.193.51:43902) 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2340793681 learned_addr learned my addr 10.20.193.51:0/2340793681 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 10 monclient(hunting): get_auth_request con 0x7f3e5815f690 auth_method 0 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7f3e56ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd220004480 con 0x7fd2300cdb20 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: dump: 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.699 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd220014070 con 0x7fd2300cdb20 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd220002920 con 0x7fd2300cdb20 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43910/0 (socket says 10.20.193.51:43910) 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 1 -- 10.20.193.51:0/1044398007 learned_addr learned my addr 10.20.193.51:0/1044398007 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 10 monclient(hunting): get_auth_request con 0x7fe24415c790 auth_method 0 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd23702b640 5 monclient: authenticate success, global_id 4486 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd226ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fe2492b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd23702b640 10 monclient: _renew_subs 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd23702b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd23014ae90 con 0x7fd2300cdb20 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fd2257fa640 10 monclient: dump: 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fb5d77fe640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80099d0 tx=0x7fb5c80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.686+0000 7fb5d4ff9640 10 client.?.objecter ms_handle_connect 0x7fb5d81526a0 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fb5d4ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 monclient: _renew_subs 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd23014b450 con 0x7fd2300cdb20 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 1 librados: init done 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 10 monclient: monmap: 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 librados: starting msgr at 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 librados: starting objecter 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 Processor -- start 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd230160600 con 0x7fd2300cdb20 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 -- start start 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 librados: setting wanted keys 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 1 librados: calling monclient init 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f7d55232640 10 monclient: init 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.687+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 0x7f7d501f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-a con 0x7f7d501526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7d50152a80 0x7f7d500ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-b con 0x7f7d50152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 0x7f7d50159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5015d3b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500748e0 con 0x7f7d501526a0 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f7d50074c50 con 0x7f7d50152a80 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7d500cdcf0 con 0x7f7d5015d3b0 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7f7d55232640 10 monclient(hunting): authenticate will time out at 1134.268295s 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.688+0000 7fe2492b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2340793681 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3e58155b50 msgr2=0x7f3e580c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3e58155b50 0x7f3e580c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3e5815fa70 msgr2=0x7f3e581e9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3e5815fa70 0x7f3e581e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.690+0000 7f3e56ffd640 1 -- 10.20.193.51:0/2340793681 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e581f41b0 con 0x7f3e5815f690 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 10 monclient(hunting): handle_auth_done global_id 4478 payload 824 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 -- 10.20.193.51:0/1044398007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe244155e60 msgr2=0x7fe24413eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe244155e60 0x7fe24413eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 -- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe2440c9470 msgr2=0x7fe2441499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe2440c9470 0x7fe2441499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 monclient: found mon.noname-c 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.692+0000 7fe2492b6640 1 -- 10.20.193.51:0/1044398007 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe244149ee0 con 0x7fe24415c790 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8046020 con 0x7fb5d81526a0 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: dump: 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.705 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5c8042640 con 0x7fb5d81526a0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c803c040 con 0x7fb5d81526a0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d4ff9640 10 monclient: dump: 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5d67fc640 4 set_mon_vals no callback set 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5de6f3640 5 monclient: authenticate success, global_id 4472 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.693+0000 7fb5de6f3640 10 monclient: _renew_subs 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d8157920 con 0x7fb5d81526a0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 monclient: _renew_subs 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d80d1160 con 0x7fb5d81526a0 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 1 librados: init done 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d80bb6f0 con 0x7fb5d81526a0 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238004cf0 con 0x7f4230004900 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: dump: 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238007110 con 0x7f4230004900 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238007500 con 0x7f4230004900 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 5 monclient: authenticate success, global_id 4503 2026-02-22T21:37:38.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42237fe640 10 monclient: dump: 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f4241573640 4 set_mon_vals no callback set 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 10 monclient: _renew_subs 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42441f7670 con 0x7f4230004900 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 10 monclient: _renew_subs 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42441f3640 con 0x7f4230004900 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.694+0000 7f42490c6640 1 librados: init done 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f3e56ffd640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e4802f310 tx=0x7f3e480047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f3e33fff640 10 client.?.objecter ms_handle_connect 0x7f3e5815f690 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f3e33fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7fe2492b6640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe240009e50 tx=0x7fe240004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7fe2397fa640 10 client.?.objecter ms_handle_connect 0x7fe24415c790 2026-02-22T21:37:38.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7fe2397fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f42490c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f42490c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f42490c6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.695+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424415cbd0 con 0x7f4230004900 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7f7d4f577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 0x7f7d50159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240046020 con 0x7fe24415c790 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 10 monclient: dump: 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe2397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe24ad40640 5 monclient: authenticate success, global_id 4478 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe24ad40640 10 monclient: _renew_subs 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe24ad40640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.697+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2441f3130 con 0x7fe24415c790 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 monclient: _renew_subs 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2441f40d0 con 0x7fe24415c790 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 1 librados: init done 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.698+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe24415cbd0 con 0x7fe24415c790 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 0x7f7d50159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43926/0 (socket says 10.20.193.51:43926) 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 1 -- 10.20.193.51:0/3546517530 learned_addr learned my addr 10.20.193.51:0/3546517530 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 10 monclient(hunting): get_auth_request con 0x7f7d5015d3b0 auth_method 0 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.701+0000 7f7d4f577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.726+0000 7f8a81fd5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:38.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.726+0000 7f8a81fd5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:38.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.726+0000 7f8a81fd5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.726+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8a7c159660 con 0x7f8a7c156040 2026-02-22T21:37:38.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.757+0000 7f53bc798640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:38.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.757+0000 7f53bc798640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:38.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.757+0000 7f53bc798640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.757+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f53b40747b0 con 0x7f53b415b930 2026-02-22T21:37:38.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.787+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fbe7401d030 con 0x7fbe8c155de0 2026-02-22T21:37:38.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.788+0000 7fbe817fa640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 0x7fbe5c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.788+0000 7fbe90f8f640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 0x7fbe5c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.788+0000 7fbe90f8f640 10 monclient: get_auth_request con 0x7fbe5c078e50 auth_method 0 2026-02-22T21:37:38.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.788+0000 7fbe90f8f640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 0x7fbe5c07b310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.788+0000 7fbe817fa640 10 client.4489.objecter ms_handle_connect 0x7fbe5c078e50 2026-02-22T21:37:38.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.789+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fbe7405d340 con 0x7fbe8c155de0 2026-02-22T21:37:38.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.789+0000 7fbe817fa640 10 client.4489.objecter ms_dispatch2 0x7fbe8c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.789+0000 7fbe817fa640 3 client.4489.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.789+0000 7fbe817fa640 3 client.4489.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.789+0000 7fbe817fa640 20 client.4489.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.796+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fbe8c0cbee0 con 0x7fbe8c155de0 2026-02-22T21:37:38.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.796+0000 7fbe817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.796+0000 7fbe817fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 1 -- 10.20.193.51:0/1988505003 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c03c070 con 0x7f6f1c1526a0 2026-02-22T21:37:38.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: dump: 2026-02-22T21:37:38.800 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 1 -- 10.20.193.51:0/1988505003 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0c02fb70 con 0x7f6f1c1526a0 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 1 -- 10.20.193.51:0/1988505003 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c037440 con 0x7f6f1c1526a0 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f20cb3640 10 monclient: dump: 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.802 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f23f40640 5 monclient: authenticate success, global_id 4480 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.797+0000 7f6f23f40640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 4 set_mon_vals no callback set 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 10 monclient: shutdown 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 -- 10.20.193.51:0/1988505003 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 msgr2=0x7f6f1c0733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 --2- 10.20.193.51:0/1988505003 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c0733d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0098a0 tx=0x7f6f0c02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 -- 10.20.193.51:0/1988505003 shutdown_connections 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 --2- 10.20.193.51:0/1988505003 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c0733d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 -- 10.20.193.51:0/1988505003 >> 10.20.193.51:0/1988505003 conn(0x7f6f1c0ce350 msgr2=0x7f6f1c0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 -- 10.20.193.51:0/1988505003 shutdown_connections 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.798+0000 7f6f23f40640 1 -- 10.20.193.51:0/1988505003 wait complete. 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5868012070 con 0x7f5878155c40 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: dump: 2026-02-22T21:37:38.804 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58680044a0 con 0x7f5878155c40 2026-02-22T21:37:38.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58680029c0 con 0x7f5878155c40 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f58667fc640 10 monclient: dump: 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f5867fff640 4 set_mon_vals no callback set 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 5 monclient: authenticate success, global_id 4483 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _renew_subs 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58781e8700 con 0x7f5878155c40 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 10 monclient: monmap: 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.807 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _renew_subs 2026-02-22T21:37:38.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58781e8cc0 con 0x7f5878155c40 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 1 librados: init done 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 librados: starting msgr at 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 librados: starting objecter 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 Processor -- start 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 -- start start 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 librados: setting wanted keys 2026-02-22T21:37:38.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 1 librados: calling monclient init 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f6f23f40640 10 monclient: init 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8007dc0 con 0x7f28e0003ba0 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: dump: 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.810 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8005bf0 con 0x7f28e0003ba0 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 1 -- 10.20.193.51:0/986871263 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8007070 con 0x7f28e0003ba0 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f97b2640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f97b2640 4 set_mon_vals no callback set 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f28f1ffb640 10 monclient: dump: 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.812 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.799+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f587806dd00 con 0x7f5878155c40 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c1f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-b con 0x7f6f1c1526a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1c152a80 0x7f6f1c1590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-c con 0x7f6f1c152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): picked mon.noname-a con 0x7f6f1c15d3b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0748e0 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c074c50 con 0x7f6f1c1526a0 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0cdcf0 con 0x7f6f1c152a80 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f6f23f40640 10 monclient(hunting): authenticate will time out at 1134.380105s 2026-02-22T21:37:38.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 10 monclient: shutdown 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 -- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 msgr2=0x7f28e0003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28e0003fa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c320 tx=0x7f28e802f5e0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 -- 10.20.193.51:0/986871263 shutdown_connections 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 --2- 10.20.193.51:0/986871263 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28e0003fa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 -- 10.20.193.51:0/986871263 >> 10.20.193.51:0/986871263 conn(0x7f28f40ce350 msgr2=0x7f28f40ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 -- 10.20.193.51:0/986871263 shutdown_connections 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 -- 10.20.193.51:0/986871263 wait complete. 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 10 monclient: build_initial_monmap 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 10 monclient: monmap: 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:38.817 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 librados: starting msgr at 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.800+0000 7f28f97b2640 1 librados: starting objecter 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 Processor -- start 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 -- start start 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 librados: setting wanted keys 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 librados: calling monclient init 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient: init 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28f41fbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-b con 0x7f28e0003ba0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:38.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28f415f570 0x7f28f40c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-c con 0x7f28f415f570 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): picked mon.noname-a con 0x7f28f415f9f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4162760 con 0x7f28f415f9f0 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4163090 con 0x7f28e0003ba0 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4162a40 con 0x7f28f415f570 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:38.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.801+0000 7f28f97b2640 10 monclient(hunting): authenticate will time out at 1134.381464s 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8004480 con 0x7fedd0155e60 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: dump: 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.820 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb8014070 con 0x7fedd0155e60 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8002920 con 0x7fedd0155e60 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedaf7fe640 10 monclient: dump: 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.821 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedcd7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 5 monclient: authenticate success, global_id 4492 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 10 monclient: _renew_subs 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd0150430 con 0x7fedd0155e60 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 10 monclient: _renew_subs 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd01509f0 con 0x7fedd0155e60 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.805+0000 7fedd52a0640 1 librados: init done 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.806+0000 7fedd52a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.806+0000 7fedd52a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.806+0000 7fedd52a0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.806+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd015cbd0 con 0x7fedd0155e60 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.807+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9e0c01d030 con 0x7f9e1c154120 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.807+0000 7f9e02ffd640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 0x7f9de807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.807+0000 7f9e1a575640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 0x7f9de807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.807+0000 7f9e1a575640 10 monclient: get_auth_request con 0x7f9de8078e50 auth_method 0 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.808+0000 7f9e1a575640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 0x7f9de807b310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e1003c000 tx=0x7f9e10037000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.808+0000 7f9e02ffd640 10 client.4500.objecter ms_handle_connect 0x7f9de8078e50 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.818+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f423800a640 con 0x7f4230004900 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.819+0000 7f42237fe640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 0x7f421007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.819+0000 7f4242575640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 0x7f421007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.819+0000 7f4242575640 10 monclient: get_auth_request con 0x7f4210078e50 auth_method 0 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.819+0000 7f4242575640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 0x7f421007b310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f422c037000 tx=0x7f422c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.819+0000 7f42237fe640 10 client.4503.objecter ms_handle_connect 0x7f4210078e50 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f9e0c00cd20 con 0x7f9e1c154120 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 10 client.4500.objecter ms_dispatch2 0x7f9e1c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 3 client.4500.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 3 client.4500.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 20 client.4500.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9e1c159fc0 con 0x7f9e1c154120 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.827+0000 7f9e02ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f42380076a0 con 0x7f4230004900 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 10 client.4503.objecter ms_dispatch2 0x7f4244000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 3 client.4503.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 3 client.4503.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 20 client.4503.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f42380625c0 con 0x7f4230004900 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.834+0000 7f42237fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8004480 con 0x7f99101526a0 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: dump: 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.842 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f8010070 con 0x7f99101526a0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8002920 con 0x7f99101526a0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 5 monclient: authenticate success, global_id 4509 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 monclient: _renew_subs 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f990d7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991014a700 con 0x7f99101526a0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f98e7fff640 10 monclient: dump: 2026-02-22T21:37:38.843 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 monclient: _renew_subs 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99101e78e0 con 0x7f99101526a0 2026-02-22T21:37:38.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 1 librados: init done 2026-02-22T21:37:38.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.840+0000 7f9915a68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.841+0000 7f9915a68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.841+0000 7f9915a68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.841+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99100733d0 con 0x7f99101526a0 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.842+0000 7f6f224b6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32874/0 (socket says 10.20.193.51:32874) 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 1 -- 10.20.193.51:0/309604190 learned_addr learned my addr 10.20.193.51:0/309604190 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 10 monclient(hunting): get_auth_request con 0x7f6f1c15d3b0 auth_method 0 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f6f224b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.843+0000 7f7d4ed76640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 0x7f7d501f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f6f224b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f6f224b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f6f224b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f7d4ed76640 10 monclient(hunting): get_auth_request con 0x7f7d501526a0 auth_method 0 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f7d4ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f7d4ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f7d4ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.844+0000 7f28f37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 10 monclient(hunting): handle_auth_done global_id 4512 payload 824 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 -- 10.20.193.51:0/309604190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1c152a80 msgr2=0x7f6f1c1590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1c152a80 0x7f6f1c1590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 -- 10.20.193.51:0/309604190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 msgr2=0x7f6f1c1f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c1f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f6f224b6640 1 -- 10.20.193.51:0/309604190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c1f40f0 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.846+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32890/0 (socket says 10.20.193.51:32890) 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 1 -- 10.20.193.51:0/2384785833 learned_addr learned my addr 10.20.193.51:0/2384785833 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 10 monclient(hunting): get_auth_request con 0x7f28f415f9f0 auth_method 0 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:38.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.847+0000 7f28f37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 10 monclient(hunting): handle_auth_done global_id 4515 payload 824 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3546517530 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 msgr2=0x7f7d50159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 0x7f7d50159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3546517530 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7d50152a80 msgr2=0x7f7d500ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7d50152a80 0x7f7d500ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.848+0000 7f7d4ed76640 1 -- 10.20.193.51:0/3546517530 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d501569a0 con 0x7f7d501526a0 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.849+0000 7f28f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.849+0000 7f28f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:38.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.849+0000 7f28f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:38.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 10 monclient(hunting): handle_auth_done global_id 4518 payload 824 2026-02-22T21:37:38.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 -- 10.20.193.51:0/2384785833 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28f415f570 msgr2=0x7f28f40c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28f415f570 0x7f28f40c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 -- 10.20.193.51:0/2384785833 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 msgr2=0x7f28f41fbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:38.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28f41fbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f28f37fe640 1 -- 10.20.193.51:0/2384785833 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f40c8ce0 con 0x7f28f415f9f0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f6f224b6640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800cb40 tx=0x7f6f1800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f6f0a7fc640 10 client.?.objecter ms_handle_connect 0x7f6f1c15d3b0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f6f0a7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f7d4ed76640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 0x7f7d501f50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7d5008ea20 tx=0x7f7d440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f7d2f7fe640 10 client.?.objecter ms_handle_connect 0x7f7d501526a0 2026-02-22T21:37:38.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.852+0000 7f7d2f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.855+0000 7f28f37fe640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f28f406cfa0 tx=0x7f28e400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.855+0000 7f28d37fe640 10 client.?.objecter ms_handle_connect 0x7f28f415f9f0 2026-02-22T21:37:38.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.855+0000 7f28d37fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4403c040 con 0x7f7d501526a0 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: dump: 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.859 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d44040630 con 0x7f7d501526a0 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44040a20 con 0x7f7d501526a0 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d55232640 5 monclient: authenticate success, global_id 4515 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d4d573640 4 set_mon_vals no callback set 2026-02-22T21:37:38.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d55232640 10 monclient: _renew_subs 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d55232640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d50156c80 con 0x7f7d501526a0 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.857+0000 7f7d2f7fe640 10 monclient: dump: 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 monclient: _renew_subs 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d50157240 con 0x7f7d501526a0 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 1 librados: init done 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5015d820 con 0x7f7d501526a0 2026-02-22T21:37:38.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1800ad50 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: dump: 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18012400 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 5 monclient: authenticate success, global_id 4512 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 10 monclient: _renew_subs 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1c1f43d0 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f180127b0 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0bfff640 4 set_mon_vals no callback set 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f0a7fc640 10 monclient: dump: 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.865 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 10 monclient: _renew_subs 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1c1f4990 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.858+0000 7f6f23f40640 1 librados: init done 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.859+0000 7f6f23f40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.859+0000 7f6f23f40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.859+0000 7f6f23f40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.859+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f1c1605c0 con 0x7f6f1c15d3b0 2026-02-22T21:37:38.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.884+0000 7fbe9321a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:38.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.884+0000 7fbe9321a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:38.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.884+0000 7fbe9321a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.884+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbe8c0cc210 con 0x7fbe8c155de0 2026-02-22T21:37:38.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.902+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb5c803f030 con 0x7fb5d81526a0 2026-02-22T21:37:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.902+0000 7fb5d4ff9640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 0x7fb5a807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.902+0000 7fb5d7fff640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 0x7fb5a807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.902+0000 7fb5d7fff640 10 monclient: get_auth_request con 0x7fb5a8078e80 auth_method 0 2026-02-22T21:37:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.903+0000 7fb5d4ff9640 10 client.4472.objecter ms_handle_connect 0x7fb5a8078e80 2026-02-22T21:37:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.903+0000 7fb5d7fff640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 0x7fb5a807b340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4002c00 tx=0x7fb5c4002a50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.905+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fb5c802fa30 con 0x7fb5d81526a0 2026-02-22T21:37:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.905+0000 7fb5d4ff9640 10 client.4472.objecter ms_dispatch2 0x7fb5d8000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.905+0000 7fb5d4ff9640 3 client.4472.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.905+0000 7fb5d4ff9640 3 client.4472.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.905+0000 7fb5d4ff9640 20 client.4472.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c004480 con 0x7f0eac152a80 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: dump: 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e9c015070 con 0x7f0eac152a80 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c002920 con 0x7f0eac152a80 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 5 monclient: authenticate success, global_id 4495 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 10 monclient: _renew_subs 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eaa7fc640 4 set_mon_vals no callback set 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac14d0e0 con 0x7f0eac152a80 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0ea8ff9640 10 monclient: dump: 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 10 monclient: _renew_subs 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac14d6a0 con 0x7f0eac152a80 2026-02-22T21:37:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.906+0000 7f0eb29a9640 1 librados: init done 2026-02-22T21:37:38.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.907+0000 7f0eb29a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.907+0000 7f0eb29a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.907+0000 7f0eb29a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:38.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.907+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac0cbf20 con 0x7f0eac152a80 2026-02-22T21:37:38.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.915+0000 7f9e20e58640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:38.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.915+0000 7f9e20e58640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:38.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.916+0000 7f9e20e58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.916+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 14} v 0) -- 0x7f9e1c06d510 con 0x7f9e1c154120 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2400373f0 con 0x7fe24415c790 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240041450 con 0x7fe24415c790 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe2397fa640 10 monclient: dump: 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.920 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fe23affd640 4 set_mon_vals no callback set 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb5d80bb6f0 con 0x7fb5d81526a0 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fb5d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.919+0000 7fb5d4ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe2400415f0 con 0x7fe24415c790 2026-02-22T21:37:38.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7f42490c6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7f42490c6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7f42490c6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 12} v 0) -- 0x7f424414e740 con 0x7f4230004900 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe2397fa640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 0x7fe21407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe248ab5640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 0x7fe21407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe248ab5640 10 monclient: get_auth_request con 0x7fe214078e00 auth_method 0 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe248ab5640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 0x7fe21407b2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe234006fd0 tx=0x7fe234006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:38.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.920+0000 7fe2397fa640 10 client.4478.objecter ms_handle_connect 0x7fe214078e00 2026-02-22T21:37:38.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fe240083af0 con 0x7fe24415c790 2026-02-22T21:37:38.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7fe2397fa640 10 client.4478.objecter ms_dispatch2 0x7fe244000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:38.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7fe2397fa640 3 client.4478.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:38.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7fe2397fa640 3 client.4478.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:38.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7fe2397fa640 20 client.4478.objecter dump_active .. 0 homeless 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4803c040 con 0x7f3e5815f690 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: dump: 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.925 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e48047070 con 0x7f3e5815f690 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:38.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48042890 con 0x7f3e5815f690 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e33fff640 10 monclient: dump: 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:38.927 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e55ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 5 monclient: authenticate success, global_id 4475 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 10 monclient: _renew_subs 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e581f4490 con 0x7f3e5815f690 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 10 monclient: _renew_subs 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e581f4a50 con 0x7f3e5815f690 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.922+0000 7f3e5db55640 1 librados: init done 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.923+0000 7f3e5db55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.923+0000 7f3e5db55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.923+0000 7f3e5db55640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:38.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.923+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e580cc5d0 con 0x7f3e5815f690 2026-02-22T21:37:38.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.929+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe240050030 con 0x7fe24415c790 2026-02-22T21:37:38.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.929+0000 7fe2397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:38.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:38.929+0000 7fe2397fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.004+0000 7fbe817fa640 1 -- 10.20.193.51:0/4282826210 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbe74024080 con 0x7fbe8c155de0 2026-02-22T21:37:39.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.004+0000 7fbe817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:39.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.004+0000 7fbe817fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.007 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:39.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.005+0000 7f8a57fff640 1 -- 10.20.193.51:0/1941222871 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a68067ec0 con 0x7f8a7c156040 2026-02-22T21:37:39.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.005+0000 7f8a57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:39.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.005+0000 7f8a57fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.008 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fb5de6f3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:39.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fb5de6f3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:39.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fb5de6f3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:39.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb5d81511a0 con 0x7fb5d81526a0 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 10 librados: watch_flush enter 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 10 librados: watch_flush exit 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 20 client.4489.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 10 client.4489.objecter successfully canceled tick 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 msgr2=0x7fbe5c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 0x7fbe5c07b310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:39.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 10 monclient: shutdown 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 msgr2=0x7fbe8c0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.006+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400b780 tx=0x7fbe7400bc50 comp rx=0 tx=0).stop 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 shutdown_connections 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe5c078e50 0x7fbe5c07b310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe8c1561c0 0x7fbe8c0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe8c155de0 0x7fbe8c0ccb50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 --2- 10.20.193.51:0/4282826210 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe8c0caae0 0x7fbe8c1eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 >> 10.20.193.51:0/4282826210 conn(0x7fbe8c0d1ec0 msgr2=0x7fbe8c153600 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 shutdown_connections 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 -- 10.20.193.51:0/4282826210 wait complete. 2026-02-22T21:37:39.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7fbe9321a640 1 librados: shutdown 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 10 librados: watch_flush enter 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 10 librados: watch_flush exit 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 20 client.4474.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 10 client.4474.objecter successfully canceled tick 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 msgr2=0x7f8a4c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 0x7f8a4c07b340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c159cd0 tx=0x7f8a6c034000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 10 monclient: shutdown 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 msgr2=0x7f8a7c0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800da80 tx=0x7f8a6800df50 comp rx=0 tx=0).stop 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 shutdown_connections 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a4c078e80 0x7f8a4c07b340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a7c15b930 0x7f8a7c1efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a7c15b150 0x7f8a7c1f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 --2- 10.20.193.51:0/1941222871 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a7c156040 0x7f8a7c0c3e10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 >> 10.20.193.51:0/1941222871 conn(0x7f8a7c0cd7b0 msgr2=0x7f8a7c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 shutdown_connections 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 -- 10.20.193.51:0/1941222871 wait complete. 2026-02-22T21:37:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.007+0000 7f8a81fd5640 1 librados: shutdown 2026-02-22T21:37:39.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.013+0000 7fe24ad40640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:39.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.013+0000 7fe24ad40640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:39.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.013+0000 7fe24ad40640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:39.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.013+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 15} v 0) -- 0x7fe244153a60 con 0x7fe24415c790 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd220019460 con 0x7fd2300cdb20 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd2257fa640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 0x7fd20007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd227fff640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 0x7fd20007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd227fff640 10 monclient: get_auth_request con 0x7fd200078e80 auth_method 0 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd2257fa640 10 client.4486.objecter ms_handle_connect 0x7fd200078e80 2026-02-22T21:37:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.014+0000 7fd227fff640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 0x7fd20007b340 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd23006cf40 tx=0x7fd218031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.017 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.8 2026-02-22T21:37:39.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.6 2026-02-22T21:37:39.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.035+0000 7f539affd640 1 -- 10.20.193.51:0/3540113223 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53b0087b80 con 0x7f53b415b930 2026-02-22T21:37:39.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.035+0000 7f539affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:39.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.035+0000 7f539affd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.037 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 10 librados: watch_flush enter 2026-02-22T21:37:39.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 10 librados: watch_flush exit 2026-02-22T21:37:39.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 20 client.4454.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 10 client.4454.objecter successfully canceled tick 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 msgr2=0x7f538407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009750 tx=0x7f53a4006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 10 monclient: shutdown 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 msgr2=0x7f53b41f9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d250 tx=0x7f53b0004bb0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 shutdown_connections 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415b930 0x7f53b41f9a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b415b150 0x7f53b40737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 --2- 10.20.193.51:0/3540113223 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4156040 0x7f53b41fb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 >> 10.20.193.51:0/3540113223 conn(0x7f53b40cd7b0 msgr2=0x7f53b4158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 shutdown_connections 2026-02-22T21:37:39.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 -- 10.20.193.51:0/3540113223 wait complete. 2026-02-22T21:37:39.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.037+0000 7f53bc798640 1 librados: shutdown 2026-02-22T21:37:39.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.044+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f3e48042a30 con 0x7f3e5815f690 2026-02-22T21:37:39.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.044+0000 7f3e33fff640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 0x7f3e2807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.045+0000 7f3e577fe640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 0x7f3e2807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.045+0000 7f3e577fe640 10 monclient: get_auth_request con 0x7f3e28078e80 auth_method 0 2026-02-22T21:37:39.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.3 2026-02-22T21:37:39.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.045+0000 7f3e577fe640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 0x7f3e2807b340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e44002a30 tx=0x7f3e440079f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.045+0000 7f3e33fff640 10 client.4475.objecter ms_handle_connect 0x7f3e28078e80 2026-02-22T21:37:39.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.047+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f3e4802f9b0 con 0x7f3e5815f690 2026-02-22T21:37:39.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.047+0000 7f3e33fff640 10 client.4475.objecter ms_dispatch2 0x7f3e58000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.047+0000 7f3e33fff640 3 client.4475.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.047+0000 7f3e33fff640 3 client.4475.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.047+0000 7f3e33fff640 20 client.4475.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.053+0000 7fb5d4ff9640 1 -- 10.20.193.51:0/3831108424 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb5c8050030 con 0x7fb5d81526a0 2026-02-22T21:37:39.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.053+0000 7fb5d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:39.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.053+0000 7fb5d4ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.055 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.057 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 10 librados: watch_flush enter 2026-02-22T21:37:39.057 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 10 librados: watch_flush exit 2026-02-22T21:37:39.057 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 20 client.4472.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 10 client.4472.objecter successfully canceled tick 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 msgr2=0x7fb5a807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 0x7fb5a807b340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4002c00 tx=0x7fb5c4002a50 comp rx=0 tx=0).stop 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 10 monclient: shutdown 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 msgr2=0x7fb5d8158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80099d0 tx=0x7fb5c80047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 shutdown_connections 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5a8078e80 0x7fb5a807b340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5d8152a80 0x7fb5d81f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5d81526a0 0x7fb5d8158f90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 --2- 10.20.193.51:0/3831108424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5d80cdb20 0x7fb5d814cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 >> 10.20.193.51:0/3831108424 conn(0x7fb5d80ce350 msgr2=0x7fb5d80c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 shutdown_connections 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 -- 10.20.193.51:0/3831108424 wait complete. 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.055+0000 7fb5de6f3640 1 librados: shutdown 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.056+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f3e48088010 con 0x7f3e5815f690 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.056+0000 7f3e33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.056+0000 7f3e33fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.058+0000 7fe2397fa640 1 -- 10.20.193.51:0/1044398007 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 15}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7fe2400881b0 con 0x7fe24415c790 2026-02-22T21:37:39.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.058+0000 7fe2397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:39.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.058+0000 7fe2397fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.060 INFO:teuthology.orchestra.run.trial051.stdout:90194313221 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 10 librados: watch_flush enter 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 10 librados: watch_flush exit 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 20 client.4478.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 10 client.4478.objecter successfully canceled tick 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 msgr2=0x7fe21407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 0x7fe21407b2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe234006fd0 tx=0x7fe234006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 10 monclient: shutdown 2026-02-22T21:37:39.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 msgr2=0x7fe2441f33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe240009e50 tx=0x7fe240004b40 comp rx=0 tx=0).stop 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 shutdown_connections 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe214078e00 0x7fe21407b2c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.060+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe24415c790 0x7fe2441f33b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe244155e60 0x7fe24413eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 --2- 10.20.193.51:0/1044398007 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe2440c9470 0x7fe2441499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 >> 10.20.193.51:0/1044398007 conn(0x7fe24406d8c0 msgr2=0x7fe2440cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 shutdown_connections 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 -- 10.20.193.51:0/1044398007 wait complete. 2026-02-22T21:37:39.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.061+0000 7fe24ad40640 1 librados: shutdown 2026-02-22T21:37:39.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.0 2026-02-22T21:37:39.070 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313222 got 90194313221 for osd.15 2026-02-22T21:37:39.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.115+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fd22005ddb0 con 0x7fd2300cdb20 2026-02-22T21:37:39.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.115+0000 7fd2257fa640 10 client.4486.objecter ms_dispatch2 0x7fd230000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.115+0000 7fd2257fa640 3 client.4486.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.115+0000 7fd2257fa640 3 client.4486.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.115+0000 7fd2257fa640 20 client.4486.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.122+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd230160600 con 0x7fd2300cdb20 2026-02-22T21:37:39.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.122+0000 7fd2257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.122+0000 7fd2257fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.139+0000 7f3e5db55640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:39.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.139+0000 7f3e5db55640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:39.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.139+0000 7f3e5db55640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:39.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.139+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3e58159850 con 0x7f3e5815f690 2026-02-22T21:37:39.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.142+0000 7f3e33fff640 1 -- 10.20.193.51:0/2340793681 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e48087910 con 0x7f3e5815f690 2026-02-22T21:37:39.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.142+0000 7f3e33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:39.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.142+0000 7f3e33fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.144 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 10 librados: watch_flush enter 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 10 librados: watch_flush exit 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 20 client.4475.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 10 client.4475.objecter successfully canceled tick 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 msgr2=0x7f3e2807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 0x7f3e2807b340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e44002a30 tx=0x7f3e440079f0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 10 monclient: shutdown 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 msgr2=0x7f3e58153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e4802f310 tx=0x7f3e480047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 shutdown_connections 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3e28078e80 0x7f3e2807b340 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3e5815fa70 0x7f3e581e9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3e5815f690 0x7f3e58153cd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 --2- 10.20.193.51:0/2340793681 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3e58155b50 0x7f3e580c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 >> 10.20.193.51:0/2340793681 conn(0x7f3e580d1b50 msgr2=0x7f3e580c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 shutdown_connections 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 -- 10.20.193.51:0/2340793681 wait complete. 2026-02-22T21:37:39.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.143+0000 7f3e5db55640 1 librados: shutdown 2026-02-22T21:37:39.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.2 2026-02-22T21:37:39.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7d4403f050 con 0x7f7d501526a0 2026-02-22T21:37:39.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d2f7fe640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 0x7f7d1c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d4e575640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 0x7f7d1c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d4e575640 10 monclient: get_auth_request con 0x7f7d1c078e00 auth_method 0 2026-02-22T21:37:39.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d2f7fe640 10 client.4515.objecter ms_handle_connect 0x7f7d1c078e00 2026-02-22T21:37:39.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.173+0000 7f7d4e575640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 0x7f7d1c07b2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7d501f18a0 tx=0x7f7d38007a70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6f18012950 con 0x7f6f1c15d3b0 2026-02-22T21:37:39.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f0a7fc640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 0x7f6eec07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f21cb5640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 0x7f6eec07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f21cb5640 10 monclient: get_auth_request con 0x7f6eec078e50 auth_method 0 2026-02-22T21:37:39.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f21cb5640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 0x7f6eec07b310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c039000 tx=0x7f6f0c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.180+0000 7f6f0a7fc640 10 client.4512.objecter ms_handle_connect 0x7f6eec078e50 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f7d44082bf0 con 0x7f7d501526a0 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 10 client.4515.objecter ms_dispatch2 0x7f7d50000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 3 client.4515.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 3 client.4515.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 20 client.4515.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7d44044020 con 0x7f7d501526a0 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.193+0000 7f7d2f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f6f1805e980 con 0x7f6f1c15d3b0 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 10 client.4512.objecter ms_dispatch2 0x7f6f1c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 3 client.4512.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 3 client.4512.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 20 client.4512.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6f18012c70 con 0x7f6f1c15d3b0 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.199+0000 7f6f0a7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4017070 con 0x7f28f415f9f0 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: dump: 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:39.203 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e400a9e0 con 0x7f28f415f9f0 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4007e60 con 0x7f28f415f9f0 2026-02-22T21:37:39.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28d37fe640 10 monclient: dump: 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:39.205 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f17fa640 4 set_mon_vals no callback set 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 5 monclient: authenticate success, global_id 4518 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 10 monclient: _renew_subs 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f40c8fc0 con 0x7f28f415f9f0 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 10 monclient: _renew_subs 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f40c9580 con 0x7f28f415f9f0 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.200+0000 7f28f97b2640 1 librados: init done 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.201+0000 7f28f97b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.201+0000 7f28f97b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.201+0000 7f28f97b2640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.201+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f4162d20 con 0x7f28f415f9f0 2026-02-22T21:37:39.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.208+0000 7fd23702b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:39.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.208+0000 7fd23702b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:39.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.208+0000 7fd23702b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:39.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.208+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd2300cbf20 con 0x7fd2300cdb20 2026-02-22T21:37:39.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.209+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f98f801d030 con 0x7f99101526a0 2026-02-22T21:37:39.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.210+0000 7f98e7fff640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 0x7f98e007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.210+0000 7f990effd640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 0x7f98e007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.210+0000 7f990effd640 10 monclient: get_auth_request con 0x7f98e0078e50 auth_method 0 2026-02-22T21:37:39.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.210+0000 7f990effd640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 0x7f98e007b310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f991008f0d0 tx=0x7f9904031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.210+0000 7f98e7fff640 10 client.4509.objecter ms_handle_connect 0x7f98e0078e50 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f98f801de30 con 0x7f99101526a0 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 10 client.4509.objecter ms_dispatch2 0x7f9910000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 3 client.4509.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 3 client.4509.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 20 client.4509.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f98f8023020 con 0x7f99101526a0 2026-02-22T21:37:39.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.220+0000 7f98e7fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.228+0000 7f42237fe640 1 -- 10.20.193.51:0/2992412763 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 12}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4238061ec0 con 0x7f4230004900 2026-02-22T21:37:39.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.228+0000 7f42237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:39.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.228+0000 7f42237fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.230 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:39.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 10 librados: watch_flush enter 2026-02-22T21:37:39.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 10 librados: watch_flush exit 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 20 client.4503.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 10 client.4503.objecter successfully canceled tick 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 msgr2=0x7f421007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 0x7f421007b310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f422c037000 tx=0x7f422c031000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 10 monclient: shutdown 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 msgr2=0x7f424406df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f423800c860 tx=0x7f423800cd30 comp rx=0 tx=0).stop 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 shutdown_connections 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4210078e50 0x7f421007b310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4244150100 0x7f42440d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f424406e450 0x7f42440d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 --2- 10.20.193.51:0/2992412763 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4230004900 0x7f424406df10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 >> 10.20.193.51:0/2992412763 conn(0x7f424406d8c0 msgr2=0x7f424414ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 shutdown_connections 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 -- 10.20.193.51:0/2992412763 wait complete. 2026-02-22T21:37:39.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.230+0000 7f42490c6640 1 librados: shutdown 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.232+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f5868021480 con 0x7f5878155c40 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.232+0000 7f58667fc640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 0x7f584807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.232+0000 7f587d33e640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 0x7f584807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.232+0000 7f587d33e640 10 monclient: get_auth_request con 0x7f5848078e50 auth_method 0 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.233+0000 7f587d33e640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 0x7f584807b310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58780c5560 tx=0x7f586c006cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.233+0000 7f58667fc640 10 client.4483.objecter ms_handle_connect 0x7f5848078e50 2026-02-22T21:37:39.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.234+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f586805da00 con 0x7f5878155c40 2026-02-22T21:37:39.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.234+0000 7f58667fc640 10 client.4483.objecter ms_dispatch2 0x7f5878000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.234+0000 7f58667fc640 3 client.4483.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.234+0000 7f58667fc640 3 client.4483.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.234+0000 7f58667fc640 20 client.4483.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.12 2026-02-22T21:37:39.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.241+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f58680620c0 con 0x7f5878155c40 2026-02-22T21:37:39.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.241+0000 7f58667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.241+0000 7f58667fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.249+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f0e9c010480 con 0x7f0eac152a80 2026-02-22T21:37:39.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.249+0000 7f0ea8ff9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 0x7f0e7c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.249+0000 7f0eabfff640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 0x7f0e7c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.249+0000 7f0eabfff640 10 monclient: get_auth_request con 0x7f0e7c078e50 auth_method 0 2026-02-22T21:37:39.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.250+0000 7f0ea8ff9640 10 client.4495.objecter ms_handle_connect 0x7f0e7c078e50 2026-02-22T21:37:39.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.250+0000 7f0eabfff640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 0x7f0e7c07b310 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009750 tx=0x7f0e98006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.251+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f0e9c05e230 con 0x7f0eac152a80 2026-02-22T21:37:39.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.251+0000 7f0ea8ff9640 10 client.4495.objecter ms_dispatch2 0x7f0eac000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.251+0000 7f0ea8ff9640 3 client.4495.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.251+0000 7f0ea8ff9640 3 client.4495.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.251+0000 7f0ea8ff9640 20 client.4495.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.257+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0eac0cbf20 con 0x7f0eac152a80 2026-02-22T21:37:39.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.257+0000 7f0ea8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.257+0000 7f0ea8ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.265+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fedb8019460 con 0x7fedd0155e60 2026-02-22T21:37:39.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.266+0000 7fedaf7fe640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 0x7fed9c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.266+0000 7fedceffd640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 0x7fed9c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.266+0000 7fedceffd640 10 monclient: get_auth_request con 0x7fed9c078e50 auth_method 0 2026-02-22T21:37:39.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.266+0000 7fedaf7fe640 10 client.4492.objecter ms_handle_connect 0x7fed9c078e50 2026-02-22T21:37:39.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.266+0000 7fedceffd640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 0x7fed9c07b310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006fd0 tx=0x7fedc4006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.268+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fedb805de60 con 0x7fedd0155e60 2026-02-22T21:37:39.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.268+0000 7fedaf7fe640 10 client.4492.objecter ms_dispatch2 0x7fedd0000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.268+0000 7fedaf7fe640 3 client.4492.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.268+0000 7fedaf7fe640 3 client.4492.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.268+0000 7fedaf7fe640 20 client.4492.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.274+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fedb8062520 con 0x7fedd0155e60 2026-02-22T21:37:39.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.275+0000 7fedaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.275+0000 7fedaf7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.276+0000 7fd2257fa640 1 -- 10.20.193.51:0/334719626 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd2300cbf20 con 0x7fd2300cdb20 2026-02-22T21:37:39.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.276+0000 7fd2257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:39.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.276+0000 7fd2257fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.278 INFO:teuthology.orchestra.run.trial051.stdout:103079215107 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 10 librados: watch_flush enter 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 10 librados: watch_flush exit 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 20 client.4486.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 10 client.4486.objecter successfully canceled tick 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 msgr2=0x7fd20007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 0x7fd20007b340 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd23006cf40 tx=0x7fd218031000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 10 monclient: shutdown 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 msgr2=0x7fd230141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd22000b500 tx=0x7fd22000b9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 shutdown_connections 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd200078e80 0x7fd20007b340 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd230152a80 0x7fd230150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd2301526a0 0x7fd23014ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 --2- 10.20.193.51:0/334719626 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2300cdb20 0x7fd230141e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 >> 10.20.193.51:0/334719626 conn(0x7fd2300ce350 msgr2=0x7fd2300cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 shutdown_connections 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.278+0000 7fd23702b640 1 -- 10.20.193.51:0/334719626 wait complete. 2026-02-22T21:37:39.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.279+0000 7fd23702b640 1 librados: shutdown 2026-02-22T21:37:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.284+0000 7f7d55232640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.284+0000 7f7d55232640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.284+0000 7f7d55232640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.284+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f7d500d1130 con 0x7f7d501526a0 2026-02-22T21:37:39.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215107 for osd.1 2026-02-22T21:37:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.290+0000 7f6f23f40640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.290+0000 7f6f23f40640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.290+0000 7f6f23f40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.290+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6f1c14fac0 con 0x7f6f1c15d3b0 2026-02-22T21:37:39.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.309+0000 7f9915a68640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:39.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.309+0000 7f9915a68640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:39.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.309+0000 7f9915a68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.309+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f9910151110 con 0x7f99101526a0 2026-02-22T21:37:39.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.327+0000 7f587fdca640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:39.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.327+0000 7f587fdca640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:39.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.327+0000 7f587fdca640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:39.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.327+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f58780cbec0 con 0x7f5878155c40 2026-02-22T21:37:39.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.330+0000 7f58667fc640 1 -- 10.20.193.51:0/3413319099 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58680619c0 con 0x7f5878155c40 2026-02-22T21:37:39.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.330+0000 7f58667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:39.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.330+0000 7f58667fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.332 INFO:teuthology.orchestra.run.trial051.stdout:98784247810 2026-02-22T21:37:39.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 10 librados: watch_flush enter 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 10 librados: watch_flush exit 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 20 client.4483.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 10 client.4483.objecter successfully canceled tick 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 msgr2=0x7f584807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 0x7f584807b310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58780c5560 tx=0x7f586c006cf0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 10 monclient: shutdown 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 msgr2=0x7f58781f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f586800b5a0 tx=0x7f586800ba70 comp rx=0 tx=0).stop 2026-02-22T21:37:39.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 shutdown_connections 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5848078e50 0x7f584807b310 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f587815e760 0x7f5878151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5878156020 0x7f5878159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 --2- 10.20.193.51:0/3413319099 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5878155c40 0x7f58781f0e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 >> 10.20.193.51:0/3413319099 conn(0x7f5878073f50 msgr2=0x7f587815e050 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 shutdown_connections 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 -- 10.20.193.51:0/3413319099 wait complete. 2026-02-22T21:37:39.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.332+0000 7f587fdca640 1 librados: shutdown 2026-02-22T21:37:39.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247810 for osd.4 2026-02-22T21:37:39.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.345+0000 7f0eb29a9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:39.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.345+0000 7f0eb29a9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:39.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.345+0000 7f0eb29a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:39.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.345+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 13} v 0) -- 0x7f0eac0c8b90 con 0x7f0eac152a80 2026-02-22T21:37:39.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.347+0000 7f0ea8ff9640 1 -- 10.20.193.51:0/4277624715 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 13}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f0e9c00d360 con 0x7f0eac152a80 2026-02-22T21:37:39.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.347+0000 7f0ea8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:39.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.347+0000 7f0ea8ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.349 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:39.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 10 librados: watch_flush enter 2026-02-22T21:37:39.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 10 librados: watch_flush exit 2026-02-22T21:37:39.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 20 client.4495.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 10 client.4495.objecter successfully canceled tick 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 msgr2=0x7f0e7c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 0x7f0e7c07b310 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009750 tx=0x7f0e98006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 10 monclient: shutdown 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 msgr2=0x7f0eac1fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00b4d0 tx=0x7f0e9c00b9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 shutdown_connections 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0e7c078e50 0x7f0e7c07b310 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0eac15d3b0 0x7f0eac1571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0eac152a80 0x7f0eac1fb850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 --2- 10.20.193.51:0/4277624715 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0eac1526a0 0x7f0eac159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 >> 10.20.193.51:0/4277624715 conn(0x7f0eac0ce350 msgr2=0x7f0eac0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 shutdown_connections 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 -- 10.20.193.51:0/4277624715 wait complete. 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.349+0000 7f0eb29a9640 1 librados: shutdown 2026-02-22T21:37:39.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.351+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f28e401b3b0 con 0x7f28f415f9f0 2026-02-22T21:37:39.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.351+0000 7f9e02ffd640 1 -- 10.20.193.51:0/2838770376 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 14}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f9e0c0620b0 con 0x7f9e1c154120 2026-02-22T21:37:39.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.351+0000 7f9e02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:39.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.351+0000 7f9e02ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.354 INFO:teuthology.orchestra.run.trial051.stdout:98784247810 2026-02-22T21:37:39.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.352+0000 7f28d37fe640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 0x7f28c007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:39.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.352+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 0x7f28c007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:39.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.352+0000 7f28f2ffd640 10 monclient: get_auth_request con 0x7f28c0078e80 auth_method 0 2026-02-22T21:37:39.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.352+0000 7f28d37fe640 10 client.4518.objecter ms_handle_connect 0x7f28c0078e80 2026-02-22T21:37:39.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.352+0000 7f28f2ffd640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 0x7f28c007b340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036000 tx=0x7f28e8007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 10 librados: watch_flush enter 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 10 librados: watch_flush exit 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 20 client.4500.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 10 client.4500.objecter successfully canceled tick 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 msgr2=0x7f9de807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 0x7f9de807b310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e1003c000 tx=0x7f9e10037000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 10 monclient: shutdown 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 msgr2=0x7f9e1c158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00ba10 tx=0x7f9e0c00bee0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 shutdown_connections 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9de8078e50 0x7f9de807b310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e1c154120 0x7f9e1c158740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e1c153d40 0x7f9e1c0cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 --2- 10.20.193.51:0/2838770376 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e1c0c8330 0x7f9e1c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 >> 10.20.193.51:0/2838770376 conn(0x7f9e1c0cb010 msgr2=0x7f9e1c0ca740 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 shutdown_connections 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 -- 10.20.193.51:0/2838770376 wait complete. 2026-02-22T21:37:39.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.354+0000 7f9e20e58640 1 librados: shutdown 2026-02-22T21:37:39.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.355+0000 7f6f0a7fc640 1 -- 10.20.193.51:0/309604190 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6f18063040 con 0x7f6f1c15d3b0 2026-02-22T21:37:39.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.355+0000 7f6f0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:39.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.355+0000 7f6f0a7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.359 INFO:teuthology.orchestra.run.trial051.stdout:94489280516 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 10 librados: watch_flush enter 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 10 librados: watch_flush exit 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 20 client.4512.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 10 client.4512.objecter successfully canceled tick 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 msgr2=0x7f6eec07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 0x7f6eec07b310 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c039000 tx=0x7f6f0c031000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 10 monclient: shutdown 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 msgr2=0x7f6f1c15c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.357+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800cb40 tx=0x7f6f1800a430 comp rx=0 tx=0).stop 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 shutdown_connections 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6eec078e50 0x7f6eec07b310 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f1c15d3b0 0x7f6f1c15c440 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1c152a80 0x7f6f1c1590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 --2- 10.20.193.51:0/309604190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f1c1526a0 0x7f6f1c1f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 >> 10.20.193.51:0/309604190 conn(0x7f6f1c0ce350 msgr2=0x7f6f1c0cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 shutdown_connections 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 -- 10.20.193.51:0/309604190 wait complete. 2026-02-22T21:37:39.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.358+0000 7f6f23f40640 1 librados: shutdown 2026-02-22T21:37:39.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247812 for osd.13 2026-02-22T21:37:39.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f28e400add0 con 0x7f28f415f9f0 2026-02-22T21:37:39.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 10 client.4518.objecter ms_dispatch2 0x7f28f4000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:39.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 3 client.4518.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:39.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 3 client.4518.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:39.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 20 client.4518.objecter dump_active .. 0 homeless 2026-02-22T21:37:39.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f28e4021020 con 0x7f28f415f9f0 2026-02-22T21:37:39.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:39.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.362+0000 7f28d37fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:39.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.14 2026-02-22T21:37:39.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.365+0000 7fedd52a0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:39.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.365+0000 7fedd52a0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:39.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.365+0000 7fedd52a0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:39.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.365+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fedd014ee20 con 0x7fedd0155e60 2026-02-22T21:37:39.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.7 2026-02-22T21:37:39.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.367+0000 7fedaf7fe640 1 -- 10.20.193.51:0/445221448 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fedb8061e20 con 0x7fedd0155e60 2026-02-22T21:37:39.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.367+0000 7fedaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:39.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.367+0000 7fedaf7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.369 INFO:teuthology.orchestra.run.trial051.stdout:98784247810 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 10 librados: watch_flush enter 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 10 librados: watch_flush exit 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 20 client.4492.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 10 client.4492.objecter successfully canceled tick 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 msgr2=0x7fed9c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 0x7fed9c07b310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006fd0 tx=0x7fedc4006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 10 monclient: shutdown 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 msgr2=0x7fedd0148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b500 tx=0x7fedb800b9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 shutdown_connections 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fed9c078e50 0x7fed9c07b310 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fedd015c790 0x7fedd0147520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fedd0155e60 0x7fedd0148d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 --2- 10.20.193.51:0/445221448 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fedd00c9470 0x7fedd0151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 >> 10.20.193.51:0/445221448 conn(0x7fedd006d8c0 msgr2=0x7fedd01546a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 shutdown_connections 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 -- 10.20.193.51:0/445221448 wait complete. 2026-02-22T21:37:39.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.369+0000 7fedd52a0640 1 librados: shutdown 2026-02-22T21:37:39.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.375+0000 7f7d2f7fe640 1 -- 10.20.193.51:0/3546517530 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f7d440872b0 con 0x7f7d501526a0 2026-02-22T21:37:39.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.375+0000 7f7d2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:39.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.375+0000 7f7d2f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.377 INFO:teuthology.orchestra.run.trial051.stdout:94489280516 2026-02-22T21:37:39.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.9 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 10 librados: watch_flush enter 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 10 librados: watch_flush exit 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 20 client.4515.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 10 client.4515.objecter successfully canceled tick 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 msgr2=0x7f7d1c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 0x7f7d1c07b2c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7d501f18a0 tx=0x7f7d38007a70 comp rx=0 tx=0).stop 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 10 monclient: shutdown 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 msgr2=0x7f7d501f50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 0x7f7d501f50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7d5008ea20 tx=0x7f7d440047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 shutdown_connections 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7d1c078e00 0x7f7d1c07b2c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7d5015d3b0 0x7f7d50159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7d50152a80 0x7f7d500ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 --2- 10.20.193.51:0/3546517530 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7d501526a0 0x7f7d501f50e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 >> 10.20.193.51:0/3546517530 conn(0x7f7d500ce350 msgr2=0x7f7d5006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 shutdown_connections 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 -- 10.20.193.51:0/3546517530 wait complete. 2026-02-22T21:37:39.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f7d55232640 1 librados: shutdown 2026-02-22T21:37:39.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f98e7fff640 1 -- 10.20.193.51:0/1229746078 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f98f8061b80 con 0x7f99101526a0 2026-02-22T21:37:39.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f98e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:39.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.378+0000 7f98e7fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.383 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 10 librados: watch_flush enter 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 10 librados: watch_flush exit 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 20 client.4509.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 10 client.4509.objecter successfully canceled tick 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 msgr2=0x7f98e007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 0x7f98e007b310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f991008f0d0 tx=0x7f9904031000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 10 monclient: shutdown 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 msgr2=0x7f99101419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b780 tx=0x7f98f800bc50 comp rx=0 tx=0).stop 2026-02-22T21:37:39.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 shutdown_connections 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f98e0078e50 0x7f98e007b310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9910152a80 0x7f99101e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f99101526a0 0x7f99101419e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 --2- 10.20.193.51:0/1229746078 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f99100cdb20 0x7f99101f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 >> 10.20.193.51:0/1229746078 conn(0x7f99100ce350 msgr2=0x7f99100ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 shutdown_connections 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 -- 10.20.193.51:0/1229746078 wait complete. 2026-02-22T21:37:39.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.380+0000 7f9915a68640 1 librados: shutdown 2026-02-22T21:37:39.388 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.11 2026-02-22T21:37:39.390 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.10 2026-02-22T21:37:39.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.447+0000 7f28f97b2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:39.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.447+0000 7f28f97b2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:39.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.447+0000 7f28f97b2640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:39.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.447+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f28f4163410 con 0x7f28f415f9f0 2026-02-22T21:37:39.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.450+0000 7f28d37fe640 1 -- 10.20.193.51:0/2384785833 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f28e4061d30 con 0x7f28f415f9f0 2026-02-22T21:37:39.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.450+0000 7f28d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:39.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.450+0000 7f28d37fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:39.452 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:39.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 10 librados: watch_flush enter 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 10 librados: watch_flush exit 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 20 client.4518.objecter shutdown clearing up homeless session... 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 10 client.4518.objecter successfully canceled tick 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 msgr2=0x7f28c007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 0x7f28c007b340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f28e8036000 tx=0x7f28e8007000 comp rx=0 tx=0).stop 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 10 monclient: shutdown 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 msgr2=0x7f28f40ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f28f406cfa0 tx=0x7f28e400cce0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 shutdown_connections 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f28c0078e80 0x7f28c007b340 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f28f415f9f0 0x7f28f40ca2c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f28f415f570 0x7f28f40c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 --2- 10.20.193.51:0/2384785833 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f28e0003ba0 0x7f28f41fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 >> 10.20.193.51:0/2384785833 conn(0x7f28f40ce350 msgr2=0x7f28f4160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 shutdown_connections 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 -- 10.20.193.51:0/2384785833 wait complete. 2026-02-22T21:37:39.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:39.452+0000 7f28f97b2640 1 librados: shutdown 2026-02-22T21:37:39.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.5 2026-02-22T21:37:40.018 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-22T21:37:40.019 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-22T21:37:40.048 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T21:37:40.066 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T21:37:40.070 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2026-02-22T21:37:40.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 10 monclient: monmap: 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.086929+0000 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.086929+0000 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.088 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.089 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 10 monclient: monmap: 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.087068+0000 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.087068+0000 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.090 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.091 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 1 Processor -- start 2026-02-22T21:37:40.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 1 -- start start 2026-02-22T21:37:40.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7efd69b03640 10 monclient: init 2026-02-22T21:37:40.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 1 Processor -- start 2026-02-22T21:37:40.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 1 -- start start 2026-02-22T21:37:40.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.086+0000 7f718ffff640 10 monclient: init 2026-02-22T21:37:40.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient: _add_conns ranks=[2,0,1,3,4,5] 2026-02-22T21:37:40.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd641526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd64152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7efd6415d3b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64073ae0 con 0x7efd64152a80 2026-02-22T21:37:40.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64073d60 con 0x7efd6415d3b0 2026-02-22T21:37:40.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64074040 con 0x7efd641526a0 2026-02-22T21:37:40.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7efd69b03640 10 monclient(hunting): authenticate will time out at 1135.667538s 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient: _add_conns ranks=[0,1,4,3,5,2] 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7190156010 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f71900caca0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f71900c91f0 0x7f7190166660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f71900c91f0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71900c9920 con 0x7f7190156010 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71900c9c00 con 0x7f71900caca0 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f71900cc770 con 0x7f71900c91f0 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.087+0000 7f718ffff640 10 monclient(hunting): authenticate will time out at 1135.667649s 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.088+0000 7efd62ffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7efd64152a80 0x7efd640cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47022/0 (socket says 10.20.193.51:47022) 2026-02-22T21:37:40.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.088+0000 7efd637fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7efd641526a0 0x7efd640733d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33152/0 (socket says 10.20.193.51:33152) 2026-02-22T21:37:40.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.088+0000 7efd62ffd640 1 -- 10.20.193.51:0/792363405 learned_addr learned my addr 10.20.193.51:0/792363405 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.088+0000 7f718e7fc640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f71900caca0 0x7f71900c8ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54450/0 (socket says 10.20.193.51:54450) 2026-02-22T21:37:40.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.088+0000 7f718e7fc640 1 -- 10.20.193.51:0/2410531690 learned_addr learned my addr 10.20.193.51:0/2410531690 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7f718f7fe640 1 --2- 10.20.193.51:0/2410531690 >> v2:10.20.193.68:3300/0 conn(0x7f71900c91f0 0x7f7190166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3587665968 0 0) 0x7efd64074040 con 0x7efd641526a0 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 10 monclient(hunting): my global_id is 4484 2026-02-22T21:37:40.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd48003620 con 0x7efd641526a0 2026-02-22T21:37:40.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7f718f7fe640 10 monclient(hunting): get_auth_request con 0x7f71900c91f0 auth_method 0 2026-02-22T21:37:40.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7f718f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7f718f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.089+0000 7f718f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740718599 0 0) 0x7efd54002920 con 0x7efd64152a80 2026-02-22T21:37:40.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 10 monclient(hunting): my global_id is 4521 2026-02-22T21:37:40.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd64074040 con 0x7efd64152a80 2026-02-22T21:37:40.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 350879650 0 0) 0x7f71900c9920 con 0x7f7190156010 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 10 monclient(hunting): my global_id is 4524 2026-02-22T21:37:40.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.090+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f717c003620 con 0x7f7190156010 2026-02-22T21:37:40.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161511159 0 0) 0x7f71900c9c00 con 0x7f71900caca0 2026-02-22T21:37:40.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 10 monclient(hunting): my global_id is 4504 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 913292642 0 0) 0x7efd48003620 con 0x7efd641526a0 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 >> v1:10.20.193.68:6789/0 conn(0x7efd6415d3b0 legacy=0x7efd6415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 >> v1:10.20.193.51:6789/0 conn(0x7efd64152a80 legacy=0x7efd640cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd64074320 con 0x7efd641526a0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7efd50002f00 con 0x7efd641526a0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd69b03640 5 monclient: authenticate success, global_id 4484 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd69b03640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: dump: 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.108 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 >> v1:10.20.193.125:6789/0 conn(0x7efd641526a0 legacy=0x7efd640733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd480041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): picked mon.c con 0x7efd48003dd0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd48004900 0x7efd48004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): picked mon.b con 0x7efd48004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd48005510 0x7efd4800bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): picked mon.a con 0x7efd48005510 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7efd4800c420 con 0x7efd48005510 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7efd4800c5d0 con 0x7efd48004900 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7efd4800c7d0 con 0x7efd48003dd0 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd627fc640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:40.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7efd69b03640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.091+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71900c9920 con 0x7f71900caca0 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 10 monclient(hunting): handle_auth_done global_id 4501 payload 274 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 1 -- 10.20.193.51:0/2410531690 >> v1:10.20.193.68:6789/0 conn(0x7f71900caca0 legacy=0x7f71900c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 1 -- 10.20.193.51:0/2410531690 >> v1:10.20.193.51:6789/0 conn(0x7f7190156010 legacy=0x7f71900cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.092+0000 7f718f7fe640 1 -- 10.20.193.51:0/2410531690 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71900c9ee0 con 0x7f71900c91f0 2026-02-22T21:37:40.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd480041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): get_auth_request con 0x7efd48003dd0 auth_method 0 2026-02-22T21:37:40.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.093+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd62ffd640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd48004900 0x7efd48004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 10 monclient(hunting): handle_auth_done global_id 4484 payload 293 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd63fff640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd48005510 0x7efd4800bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 -- 10.20.193.51:0/792363405 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd48004900 msgr2=0x7efd48004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd48004900 0x7efd48004d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 -- 10.20.193.51:0/792363405 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd48005510 msgr2=0x7efd4800bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd48005510 0x7efd4800bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 monclient: found mon.c 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd63fff640 1 -- 10.20.193.51:0/792363405 reap_dead start 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 1 -- 10.20.193.51:0/792363405 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd48003620 con 0x7efd48003dd0 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7efd637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.094+0000 7f718f7fe640 1 --2- 10.20.193.51:0/2410531690 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f7190166660 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7180009d90 tx=0x7f718002f230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b770a6afabe73d0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.095+0000 7efd637fe640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd480041d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c190 tx=0x7efd5002f4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491362238ad3da51 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50007c80 con 0x7efd48003dd0 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: dump: 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.119 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50033030 con 0x7efd48003dd0 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 1 -- 10.20.193.51:0/792363405 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50005b50 con 0x7efd48003dd0 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 4 set_mon_vals no callback set 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd627fc640 10 monclient: dump: 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.121 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 10 monclient: shutdown 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 -- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 msgr2=0x7efd480041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd480041d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c190 tx=0x7efd5002f4c0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 -- 10.20.193.51:0/792363405 shutdown_connections 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 --2- 10.20.193.51:0/792363405 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd480041d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 -- 10.20.193.51:0/792363405 >> 10.20.193.51:0/792363405 conn(0x7efd640ce350 msgr2=0x7efd640ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 -- 10.20.193.51:0/792363405 shutdown_connections 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.097+0000 7efd69b03640 1 -- 10.20.193.51:0/792363405 wait complete. 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 10 monclient: monmap: 2026-02-22T21:37:40.122 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 librados: starting msgr at 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 librados: starting objecter 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 Processor -- start 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 -- start start 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 librados: setting wanted keys 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 1 librados: calling monclient init 2026-02-22T21:37:40.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7efd69b03640 10 monclient: init 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718003c070 con 0x7f71900c91f0 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: dump: 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.125 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7180037440 con 0x7f71900c91f0 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 1 -- 10.20.193.51:0/2410531690 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180037810 con 0x7f71900c91f0 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718dffb640 10 monclient: dump: 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.126 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718ffff640 5 monclient: authenticate success, global_id 4501 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718ffff640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.098+0000 7f718ffff640 4 set_mon_vals no callback set 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd641f5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-c con 0x7efd48003dd0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 0x7efd641e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-a con 0x7efd6414ad00 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd641e7ff0 0x7efd64149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): picked mon.noname-b con 0x7efd641e7ff0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7efd64074c50 con 0x7efd6414ad00 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7efd640cca00 con 0x7efd641e7ff0 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7efd640748e0 con 0x7efd48003dd0 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd69b03640 10 monclient(hunting): authenticate will time out at 1135.679146s 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 10 monclient: shutdown 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 -- 10.20.193.51:0/2410531690 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 msgr2=0x7f7190166660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 --2- 10.20.193.51:0/2410531690 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f7190166660 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7180009d90 tx=0x7f718002f230 comp rx=0 tx=0).stop 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 -- 10.20.193.51:0/2410531690 shutdown_connections 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 --2- 10.20.193.51:0/2410531690 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f7190166660 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 -- 10.20.193.51:0/2410531690 >> 10.20.193.51:0/2410531690 conn(0x7f71900ce340 msgr2=0x7f71900ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 -- 10.20.193.51:0/2410531690 shutdown_connections 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 1 -- 10.20.193.51:0/2410531690 wait complete. 2026-02-22T21:37:40.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 0x7efd641e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd637fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd641f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 0x7efd641e7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32910/0 (socket says 10.20.193.51:32910) 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 1 -- 10.20.193.51:0/852436486 learned_addr learned my addr 10.20.193.51:0/852436486 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 10 monclient(hunting): get_auth_request con 0x7efd6414ad00 auth_method 0 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd62ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7f718ffff640 10 monclient: monmap: 2026-02-22T21:37:40.131 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd637fe640 10 monclient(hunting): get_auth_request con 0x7efd48003dd0 auth_method 0 2026-02-22T21:37:40.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd637fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd637fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.099+0000 7efd63fff640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd641e7ff0 0x7efd64149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 librados: starting msgr at 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 librados: starting objecter 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 Processor -- start 2026-02-22T21:37:40.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): get_auth_request con 0x7efd641e7ff0 auth_method 0 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 -- start start 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 librados: setting wanted keys 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 librados: calling monclient init 2026-02-22T21:37:40.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient: init 2026-02-22T21:37:40.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7efd63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f71901f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f71900c91f0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 0x7f7190149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f71900caca0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7190156010 0x7f71900c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f7190156010 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f71900ca810 con 0x7f7190156010 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f71900c96d0 con 0x7f71900c91f0 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f71900ca4a0 con 0x7f71900caca0 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.100+0000 7f718ffff640 10 monclient(hunting): authenticate will time out at 1135.680894s 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 10 monclient(hunting): handle_auth_done global_id 4487 payload 824 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 -- 10.20.193.51:0/852436486 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd641e7ff0 msgr2=0x7efd64149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd641e7ff0 0x7efd64149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 -- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 msgr2=0x7efd641e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 0x7efd641e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.101+0000 7efd637fe640 1 -- 10.20.193.51:0/852436486 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd641e8470 con 0x7efd48003dd0 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.102+0000 7f718e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 0x7f7190149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7190156010 0x7f71900c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 0x7f7190149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43978/0 (socket says 10.20.193.51:43978) 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 1 -- 10.20.193.51:0/2039482674 learned_addr learned my addr 10.20.193.51:0/2039482674 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 10 monclient(hunting): get_auth_request con 0x7f71900caca0 auth_method 0 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f71901f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7efd637fe640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd641f5670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efd50033000 tx=0x7efd640bbb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7efd3ffff640 10 client.?.objecter ms_handle_connect 0x7efd48003dd0 2026-02-22T21:37:40.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7efd3ffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): get_auth_request con 0x7f71900c91f0 auth_method 0 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718f7fe640 10 monclient(hunting): get_auth_request con 0x7f7190156010 auth_method 0 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.103+0000 7f718effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.104+0000 7f718e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 10 monclient(hunting): handle_auth_done global_id 4510 payload 824 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 -- 10.20.193.51:0/2039482674 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 msgr2=0x7f7190149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 0x7f7190149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 -- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7190156010 msgr2=0x7f71900c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7190156010 0x7f71900c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.105+0000 7f718effd640 1 -- 10.20.193.51:0/2039482674 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71901f7f90 con 0x7f71900c91f0 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.106+0000 7f718effd640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f71901f7a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f718400c4d0 tx=0x7f718400c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.106+0000 7f716f7fe640 10 client.?.objecter ms_handle_connect 0x7f71900c91f0 2026-02-22T21:37:40.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.106+0000 7f716f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50007070 con 0x7efd48003dd0 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: dump: 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.143 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50007210 con 0x7efd48003dd0 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 5 monclient: authenticate success, global_id 4487 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _renew_subs 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd641f5fb0 con 0x7efd48003dd0 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd500075e0 con 0x7efd48003dd0 2026-02-22T21:37:40.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd3ffff640 10 monclient: dump: 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd61ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _renew_subs 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd641f6570 con 0x7efd48003dd0 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 1 librados: init done 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6415d820 con 0x7efd48003dd0 2026-02-22T21:37:40.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71840052d0 con 0x7f71900c91f0 2026-02-22T21:37:40.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7f716f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7f716f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.107+0000 7f716f7fe640 10 monclient: dump: 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184005470 con 0x7f71900c91f0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184003930 con 0x7f71900c91f0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 5 monclient: authenticate success, global_id 4510 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _renew_subs 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71901f49c0 con 0x7f71900c91f0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718d7fa640 4 set_mon_vals no callback set 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f716f7fe640 10 monclient: dump: 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.147 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _renew_subs 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71900c5fe0 con 0x7f71900c91f0 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 1 librados: init done 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.108+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7190160420 con 0x7f71900c91f0 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.129+0000 7f071a885640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.129+0000 7f071a885640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.129+0000 7f071a885640 10 monclient: monmap: 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.130163+0000 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.130163+0000 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 Processor -- start 2026-02-22T21:37:40.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 -- start start 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient: init 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient: _add_conns ranks=[5,4,2,0,3,1] 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f07141527b0 0x7f07140c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): picked mon.noname-c con 0x7f07141527b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f07140bf800 0x7f071406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): picked mon.noname-b con 0x7f07140bf800 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07140bfc90 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071415ffa0 con 0x7f07140bfc90 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f07140be810 con 0x7f07140bf800 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f0714152b90 con 0x7f07141527b0 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.130+0000 7f071a885640 10 monclient(hunting): authenticate will time out at 1135.710682s 2026-02-22T21:37:40.154 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T21:37:40.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 10 monclient: monmap: 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.160232+0000 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.160232+0000 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.161 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 1 Processor -- start 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 1 -- start start 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.159+0000 7fcca1e4c640 10 monclient: init 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7f65d2ec9640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7f65d2ec9640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7f65d2ec9640 10 monclient: monmap: 2026-02-22T21:37:40.162 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.161094+0000 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.161094+0000 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.163 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient: _add_conns ranks=[0,2,3,5,1,4] 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc9c152ab0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc9c06d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-a con 0x7fcc9c0c95a0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc9c06f1d0 con 0x7fcc9c152ab0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc9c06de50 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fcc9c06ed40 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcca1e4c640 10 monclient(hunting): authenticate will time out at 1135.740520s 2026-02-22T21:37:40.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.160+0000 7fcc9bfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 Processor -- start 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 -- start start 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient: init 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32936/0 (socket says 10.20.193.51:32936) 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 1 -- 10.20.193.51:0/1414735066 learned_addr learned my addr 10.20.193.51:0/1414735066 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 10 monclient(hunting): get_auth_request con 0x7fcc9c0c95a0 auth_method 0 2026-02-22T21:37:40.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7fcc9bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient: _add_conns ranks=[2,3,4,1,5,0] 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65cc1526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f65cc152a80 0x7f65cc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f65cc0cdb20 0x7f65cc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc0cdb20 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65cc073ae0 con 0x7f65cc1526a0 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f65cc0cca00 con 0x7f65cc152a80 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f65cc074c50 con 0x7f65cc0cdb20 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.161+0000 7f65d2ec9640 10 monclient(hunting): authenticate will time out at 1135.741662s 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.162+0000 7fcc9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.162+0000 7fcc9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.162+0000 7fcc9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.162+0000 7f65c3fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f65cc152a80 0x7f65cc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f65cc152a80 0x7f65cc0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32944/0 (socket says 10.20.193.51:32944) 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 1 -- 10.20.193.51:0/1818557330 learned_addr learned my addr 10.20.193.51:0/1818557330 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 10 monclient(hunting): get_auth_request con 0x7f65cc152a80 auth_method 0 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7f65c3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 1 -- 10.20.193.51:0/1414735066 >> v1:10.20.193.125:6789/0 conn(0x7fcc9c06d9d0 legacy=0x7fcc9c0c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 1 -- 10.20.193.51:0/1414735066 >> v1:10.20.193.51:6789/0 conn(0x7fcc9c152ab0 legacy=0x7fcc9c0c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.164+0000 7fcc9bfff640 1 -- 10.20.193.51:0/1414735066 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c06e130 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.165+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.165+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.165+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 10 monclient(hunting): handle_auth_done global_id 4539 payload 274 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 1 -- 10.20.193.51:0/1818557330 >> v2:10.20.193.68:3300/0 conn(0x7f65cc0cdb20 msgr2=0x7f65cc15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 1 --2- 10.20.193.51:0/1818557330 >> v2:10.20.193.68:3300/0 conn(0x7f65cc0cdb20 0x7f65cc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 1 -- 10.20.193.51:0/1818557330 >> v1:10.20.193.125:6789/0 conn(0x7f65cc1526a0 legacy=0x7f65cc0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.166+0000 7f65c3fff640 1 -- 10.20.193.51:0/1818557330 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc073d60 con 0x7f65cc152a80 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.167+0000 7fcc9bfff640 1 --2- 10.20.193.51:0/1414735066 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009e80 tx=0x7fcc9002f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d50108bd1cfc3e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.168+0000 7f65c3fff640 1 --2- 10.20.193.51:0/1818557330 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc152a80 0x7f65cc0cd5e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65cc08f0d0 tx=0x7f65b402ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71163eb99288ca5d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 1 -- 10.20.193.51:0/1414735066 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9002fdd0 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: dump: 2026-02-22T21:37:40.172 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 1 -- 10.20.193.51:0/1414735066 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc900373f0 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 1 -- 10.20.193.51:0/1414735066 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc900377c0 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcca1e4c640 5 monclient: authenticate success, global_id 4533 2026-02-22T21:37:40.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcca1e4c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcca1e4c640 4 set_mon_vals no callback set 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.170+0000 7fcc9a7fc640 10 monclient: dump: 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 10 monclient: shutdown 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 -- 10.20.193.51:0/1414735066 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c0c95a0 msgr2=0x7fcc9c1632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/1414735066 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009e80 tx=0x7fcc9002f360 comp rx=0 tx=0).stop 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 -- 10.20.193.51:0/1414735066 shutdown_connections 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/1414735066 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c1632e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 -- 10.20.193.51:0/1414735066 >> 10.20.193.51:0/1414735066 conn(0x7fcc9c0cd7b0 msgr2=0x7fcc9c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 -- 10.20.193.51:0/1414735066 shutdown_connections 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 1 -- 10.20.193.51:0/1414735066 wait complete. 2026-02-22T21:37:40.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 1 -- 10.20.193.51:0/1818557330 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b403c070 con 0x7f65cc152a80 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: dump: 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 1 -- 10.20.193.51:0/1818557330 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b402fbd0 con 0x7f65cc152a80 2026-02-22T21:37:40.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65d2ec9640 5 monclient: authenticate success, global_id 4539 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65d2ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 1 -- 10.20.193.51:0/1818557330 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4037440 con 0x7f65cc152a80 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65d2ec9640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65d2ec9640 4 set_mon_vals no callback set 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7f65c37fe640 10 monclient: dump: 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.177 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.171+0000 7fcca1e4c640 10 monclient: monmap: 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 librados: starting msgr at 2026-02-22T21:37:40.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 librados: starting objecter 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 10 monclient: shutdown 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 -- 10.20.193.51:0/1818557330 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc152a80 msgr2=0x7f65cc0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/1818557330 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc152a80 0x7f65cc0cd5e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65cc08f0d0 tx=0x7f65b402ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 Processor -- start 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 -- 10.20.193.51:0/1818557330 shutdown_connections 2026-02-22T21:37:40.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/1818557330 >> v2:10.20.193.68:3300/0 conn(0x7f65cc0cdb20 0x7f65cc15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/1818557330 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc152a80 0x7f65cc0cd5e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 -- 10.20.193.51:0/1818557330 >> 10.20.193.51:0/1818557330 conn(0x7f65cc0ce350 msgr2=0x7f65cc0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 -- 10.20.193.51:0/1818557330 shutdown_connections 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 -- start start 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 librados: setting wanted keys 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 librados: calling monclient init 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient: init 2026-02-22T21:37:40.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 -- 10.20.193.51:0/1818557330 wait complete. 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-a con 0x7fcc9c06d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-b con 0x7fcc9c0c95a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc9c152ab0 0x7fcc9c1487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): picked mon.noname-c con 0x7fcc9c152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c06e9d0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c06e6f0 con 0x7fcc9c0c95a0 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c0c61c0 con 0x7fcc9c152ab0 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcca1e4c640 10 monclient(hunting): authenticate will time out at 1135.752743s 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 10 monclient: monmap: 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.182 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7fcc9b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 librados: starting msgr at 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.172+0000 7f65d2ec9640 1 librados: starting objecter 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 Processor -- start 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 -- start start 2026-02-22T21:37:40.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 librados: setting wanted keys 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 librados: calling monclient init 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient: init 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32952/0 (socket says 10.20.193.51:32952) 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 1 -- 10.20.193.51:0/3885846711 learned_addr learned my addr 10.20.193.51:0/3885846711 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): get_auth_request con 0x7fcc9c06d9d0 auth_method 0 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7fcc9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f65cc0cdb20 0x7f65cc141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f65cc0cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f65cc152a80 0x7f65cc150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f65cc152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc074600 con 0x7f65cc1526a0 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc074320 con 0x7f65cc0cdb20 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc0748e0 con 0x7f65cc152a80 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.173+0000 7f65d2ec9640 10 monclient(hunting): authenticate will time out at 1135.753513s 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 10 monclient(hunting): handle_auth_done global_id 4542 payload 824 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 -- 10.20.193.51:0/3885846711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc9c152ab0 msgr2=0x7fcc9c1487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc9c152ab0 0x7fcc9c1487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 -- 10.20.193.51:0/3885846711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc9c0c95a0 msgr2=0x7fcc9c15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.174+0000 7fcc9b7fe640 1 -- 10.20.193.51:0/3885846711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c15a510 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7fcc9b7fe640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800c480 tx=0x7fcc8800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7fcc77fff640 10 client.?.objecter ms_handle_connect 0x7fcc9c06d9d0 2026-02-22T21:37:40.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7fcc77fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:32968/0 (socket says 10.20.193.51:32968) 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 1 -- 10.20.193.51:0/4242465856 learned_addr learned my addr 10.20.193.51:0/4242465856 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 10 monclient(hunting): get_auth_request con 0x7f65cc1526a0 auth_method 0 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.176+0000 7f65c3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.177+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.177+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.177+0000 7f65c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 10 monclient(hunting): handle_auth_done global_id 4545 payload 824 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 -- 10.20.193.51:0/4242465856 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f65cc152a80 msgr2=0x7f65cc150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f65cc152a80 0x7f65cc150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 -- 10.20.193.51:0/4242465856 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f65cc0cdb20 msgr2=0x7f65cc141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f65cc0cdb20 0x7f65cc141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7f65c3fff640 1 -- 10.20.193.51:0/4242465856 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc14abb0 con 0x7f65cc1526a0 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88017070 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: dump: 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc880052f0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc880036f0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc77fff640 10 monclient: dump: 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcc99ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 5 monclient: authenticate success, global_id 4542 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 10 monclient: _renew_subs 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c15a7f0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 10 monclient: _renew_subs 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c15adb0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.179+0000 7fcca1e4c640 1 librados: init done 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.180+0000 7fcca1e4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.180+0000 7fcca1e4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.180+0000 7fcca1e4c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.180+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9c06d790 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.182+0000 7f65c3fff640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f65b4002410 tx=0x7f65b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.182+0000 7f65c17fa640 10 client.?.objecter ms_handle_connect 0x7f65cc1526a0 2026-02-22T21:37:40.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.182+0000 7f65c17fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.213+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7184003ad0 con 0x7f71900c91f0 2026-02-22T21:37:40.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7f716f7fe640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 0x7f715c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7f718e7fc640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 0x7f715c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7f718e7fc640 10 monclient: get_auth_request con 0x7f715c078e00 auth_method 0 2026-02-22T21:37:40.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7f718e7fc640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 0x7f715c07b2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f719006e160 tx=0x7f7178009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7f716f7fe640 10 client.4510.objecter ms_handle_connect 0x7f715c078e00 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.214+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7efd5002fce0 con 0x7efd48003dd0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7efd3ffff640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 0x7efd3407b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7efd62ffd640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 0x7efd3407b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7efd62ffd640 10 monclient: get_auth_request con 0x7efd34078e30 auth_method 0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7efd3ffff640 10 client.4487.objecter ms_handle_connect 0x7efd34078e30 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7efd62ffd640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 0x7efd3407b2f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd54002770 tx=0x7efd54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f718405e000 con 0x7f71900c91f0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7f716f7fe640 10 client.4510.objecter ms_dispatch2 0x7f7190000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7f716f7fe640 3 client.4510.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.215+0000 7f716f7fe640 3 client.4510.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7f716f7fe640 20 client.4510.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7efd50082200 con 0x7efd48003dd0 2026-02-22T21:37:40.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7efd3ffff640 10 client.4487.objecter ms_dispatch2 0x7efd64000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7efd3ffff640 3 client.4487.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7efd3ffff640 3 client.4487.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.216+0000 7efd3ffff640 20 client.4487.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.221+0000 7fd7547e3640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.221+0000 7fd7547e3640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.221+0000 7fd7547e3640 10 monclient: monmap: 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.222313+0000 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.222313+0000 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.223 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 Processor -- start 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 -- start start 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient: init 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient: _add_conns ranks=[2,4,3,0,5,1] 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd74c152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd74c06d9d0 0x7fd74c0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-b con 0x7fd74c06d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd74c0c93d0 0x7fd74c158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-a con 0x7fd74c0c93d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c06f1d0 con 0x7fd74c152ab0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fd74c06ed40 con 0x7fd74c0c93d0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd74c0c9880 con 0x7fd74c06d9d0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7fd7547e3640 10 monclient(hunting): authenticate will time out at 1135.802607s 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7efd641e8470 con 0x7efd48003dd0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7efd3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7efd3ffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7190160420 con 0x7f71900c91f0 2026-02-22T21:37:40.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7f716f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.222+0000 7f716f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.223+0000 7f0713fff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f07141527b0 0x7f07140c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f07141527b0 0x7f07140c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43992/0 (socket says 10.20.193.51:43992) 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 1 -- 10.20.193.51:0/2383075511 learned_addr learned my addr 10.20.193.51:0/2383075511 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): get_auth_request con 0x7f07141527b0 auth_method 0 2026-02-22T21:37:40.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f07137fe640 1 --2- 10.20.193.51:0/2383075511 >> v2:10.20.193.68:3300/0 conn(0x7f07140bf800 0x7f071406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f07137fe640 10 monclient(hunting): get_auth_request con 0x7f07140bf800 auth_method 0 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f07137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f07137fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f07137fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.224+0000 7f0713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd74c06d9d0 0x7fd74c0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd74c06d9d0 0x7fd74c0c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35754/0 (socket says 10.20.193.51:35754) 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 1 -- 10.20.193.51:0/459614083 learned_addr learned my addr 10.20.193.51:0/459614083 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): get_auth_request con 0x7fd74c06d9d0 auth_method 0 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.225+0000 7fd751d57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 10 monclient(hunting): handle_auth_done global_id 4493 payload 274 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 1 -- 10.20.193.51:0/2383075511 >> v2:10.20.193.68:3300/0 conn(0x7f07140bf800 msgr2=0x7f071406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 1 --2- 10.20.193.51:0/2383075511 >> v2:10.20.193.68:3300/0 conn(0x7f07140bf800 0x7f071406fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 1 -- 10.20.193.51:0/2383075511 >> v1:10.20.193.125:6789/0 conn(0x7f07140bfc90 legacy=0x7f071415f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.226+0000 7f0713fff640 1 -- 10.20.193.51:0/2383075511 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07140bea60 con 0x7f07141527b0 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 10 monclient(hunting): handle_auth_done global_id 4516 payload 274 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 1 -- 10.20.193.51:0/459614083 >> v2:10.20.193.51:3300/0 conn(0x7fd74c0c93d0 msgr2=0x7fd74c158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 1 --2- 10.20.193.51:0/459614083 >> v2:10.20.193.51:3300/0 conn(0x7fd74c0c93d0 0x7fd74c158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 1 -- 10.20.193.51:0/459614083 >> v1:10.20.193.125:6789/0 conn(0x7fd74c152ab0 legacy=0x7fd74c0c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.227+0000 7fd751d57640 1 -- 10.20.193.51:0/459614083 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c06de50 con 0x7fd74c06d9d0 2026-02-22T21:37:40.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.228+0000 7f0713fff640 1 --2- 10.20.193.51:0/2383075511 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f07140c9660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07140c8f90 tx=0x7f070002ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60bdc0e81092268a server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.231 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.229+0000 7fd751d57640 1 --2- 10.20.193.51:0/459614083 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c06d9d0 0x7fd74c0c8e90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd740009970 tx=0x7fd74002efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcf502b27f36ee1c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 1 -- 10.20.193.51:0/2383075511 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070003c070 con 0x7f07141527b0 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: dump: 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.232 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 1 -- 10.20.193.51:0/2383075511 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070002fbd0 con 0x7f07141527b0 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 1 -- 10.20.193.51:0/2383075511 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700037440 con 0x7f07141527b0 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f0712ffd640 10 monclient: dump: 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f071a885640 5 monclient: authenticate success, global_id 4493 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f071a885640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.230+0000 7f071a885640 4 set_mon_vals no callback set 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 10 monclient: shutdown 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- 10.20.193.51:0/2383075511 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 msgr2=0x7f07140c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 --2- 10.20.193.51:0/2383075511 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f07140c9660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07140c8f90 tx=0x7f070002ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- 10.20.193.51:0/2383075511 shutdown_connections 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 --2- 10.20.193.51:0/2383075511 >> v2:10.20.193.68:3300/0 conn(0x7f07140bf800 0x7f071406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 --2- 10.20.193.51:0/2383075511 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f07140c9660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- 10.20.193.51:0/2383075511 >> 10.20.193.51:0/2383075511 conn(0x7f07140cd7b0 msgr2=0x7f07140cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- 10.20.193.51:0/2383075511 shutdown_connections 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- 10.20.193.51:0/2383075511 wait complete. 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 1 -- 10.20.193.51:0/459614083 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74003c070 con 0x7fd74c06d9d0 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: dump: 2026-02-22T21:37:40.235 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 1 -- 10.20.193.51:0/459614083 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd74002fbe0 con 0x7fd74c06d9d0 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 1 -- 10.20.193.51:0/459614083 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740037440 con 0x7fd74c06d9d0 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd751556640 10 monclient: dump: 2026-02-22T21:37:40.236 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 5 monclient: authenticate success, global_id 4516 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 4 set_mon_vals no callback set 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 10 monclient: monmap: 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 librados: starting msgr at 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 librados: starting objecter 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 Processor -- start 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 10 monclient: shutdown 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 1 -- 10.20.193.51:0/459614083 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c06d9d0 msgr2=0x7fd74c0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 1 --2- 10.20.193.51:0/459614083 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c06d9d0 0x7fd74c0c8e90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd740009970 tx=0x7fd74002efb0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 -- start start 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 librados: setting wanted keys 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 1 librados: calling monclient init 2026-02-22T21:37:40.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7f071a885640 10 monclient: init 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.231+0000 7fd7547e3640 1 -- 10.20.193.51:0/459614083 shutdown_connections 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 --2- 10.20.193.51:0/459614083 >> v2:10.20.193.51:3300/0 conn(0x7fd74c0c93d0 0x7fd74c158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 --2- 10.20.193.51:0/459614083 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c06d9d0 0x7fd74c0c8e90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 -- 10.20.193.51:0/459614083 >> 10.20.193.51:0/459614083 conn(0x7fd74c0cd7b0 msgr2=0x7fd74c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 -- 10.20.193.51:0/459614083 shutdown_connections 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 -- 10.20.193.51:0/459614083 wait complete. 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 10 monclient: monmap: 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.240 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.241 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.241 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f07140bf800 0x7f07140c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): picked mon.noname-a con 0x7f07140bf800 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): picked mon.noname-b con 0x7f07140bfc90 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f071414ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): picked mon.noname-c con 0x7f07141527b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f07140bf300 con 0x7f07140bf800 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f07140bf020 con 0x7f07140bfc90 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f071406d710 con 0x7f07141527b0 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f071a885640 10 monclient(hunting): authenticate will time out at 1135.812566s 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 librados: starting msgr at 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 librados: starting objecter 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 Processor -- start 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 -- start start 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 librados: setting wanted keys 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 1 librados: calling monclient init 2026-02-22T21:37:40.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7fd7547e3640 10 monclient: init 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f0718dfb640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f071414ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.232+0000 7f07137fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35760/0 (socket says 10.20.193.51:35760) 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 1 -- 10.20.193.51:0/1558587666 learned_addr learned my addr 10.20.193.51:0/1558587666 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): get_auth_request con 0x7f07140bfc90 auth_method 0 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): get_auth_request con 0x7f07141527b0 auth_method 0 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd74c06d9d0 0x7fd74c1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-a con 0x7fd74c06d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c0c93d0 0x7fd74c148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-b con 0x7fd74c0c93d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): picked mon.noname-c con 0x7fd74c152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c06e6f0 con 0x7fd74c06d9d0 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c06e410 con 0x7fd74c0c93d0 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c06e9d0 con 0x7fd74c152ab0 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7fd7547e3640 10 monclient(hunting): authenticate will time out at 1135.813153s 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f07137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.233+0000 7f0718dfb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 10 monclient(hunting): handle_auth_done global_id 4519 payload 824 2026-02-22T21:37:40.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 -- 10.20.193.51:0/1558587666 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 msgr2=0x7f071414ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f071414ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 -- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f07140bf800 msgr2=0x7f07140c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f07140bf800 0x7f07140c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7f07137fe640 1 -- 10.20.193.51:0/1558587666 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0714074e10 con 0x7f07140bfc90 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.234+0000 7fd752d59640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44020/0 (socket says 10.20.193.51:44020) 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 1 -- 10.20.193.51:0/3730088509 learned_addr learned my addr 10.20.193.51:0/3730088509 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): get_auth_request con 0x7fd74c152ab0 auth_method 0 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7f07137fe640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f070400e8c0 tx=0x7f070400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7f0710ff9640 10 client.?.objecter ms_handle_connect 0x7f07140bfc90 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7f0710ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.235+0000 7fd752d59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd751d57640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c0c93d0 0x7fd74c148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 -- 10.20.193.51:0/3730088509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c0c93d0 msgr2=0x7fd74c148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c0c93d0 0x7fd74c148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 -- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd74c06d9d0 msgr2=0x7fd74c1f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd74c06d9d0 0x7fd74c1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7fd752d59640 1 -- 10.20.193.51:0/3730088509 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c1f7ec0 con 0x7fd74c152ab0 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704017070 con 0x7f07140bfc90 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: dump: 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070400c940 con 0x7f07140bfc90 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 5 monclient: authenticate success, global_id 4519 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 10 monclient: _renew_subs 2026-02-22T21:37:40.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071415ad90 con 0x7f07140bfc90 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070400cd30 con 0x7f07140bfc90 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f0710ff9640 10 monclient: dump: 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f07127fc640 4 set_mon_vals no callback set 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 10 monclient: _renew_subs 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071415b350 con 0x7f07140bfc90 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.237+0000 7f071a885640 1 librados: init done 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7f071a885640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7f071a885640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7f071a885640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07140c9e20 con 0x7f07140bfc90 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7fd752d59640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd74800b9a0 tx=0x7fd74800be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7fd732ffd640 10 client.?.objecter ms_handle_connect 0x7fd74c152ab0 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.238+0000 7fd732ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74800c790 con 0x7fd74c152ab0 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: dump: 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd748010070 con 0x7fd74c152ab0 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74800cb90 con 0x7fd74c152ab0 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 5 monclient: authenticate success, global_id 4499 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd750d55640 4 set_mon_vals no callback set 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd732ffd640 10 monclient: dump: 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.253 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 10 monclient: _renew_subs 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd74c1f4620 con 0x7fd74c152ab0 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 10 monclient: _renew_subs 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd74c0c1170 con 0x7fd74c152ab0 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.239+0000 7fd7547e3640 1 librados: init done 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.240+0000 7fd7547e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.240+0000 7fd7547e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.240+0000 7fd7547e3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.240+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74c152e90 con 0x7fd74c152ab0 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.249+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f0704013470 con 0x7f07140bfc90 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.250+0000 7f0710ff9640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 0x7f06e407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.250+0000 7f0713fff640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 0x7f06e407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.250+0000 7f0713fff640 10 monclient: get_auth_request con 0x7f06e4078e50 auth_method 0 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.250+0000 7f0713fff640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 0x7f06e407b310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0714156280 tx=0x7f0700031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.250+0000 7f0710ff9640 10 client.4519.objecter ms_handle_connect 0x7f06e4078e50 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.251+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f070405e470 con 0x7f07140bfc90 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.251+0000 7f0710ff9640 10 client.4519.objecter ms_dispatch2 0x7f0714000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.251+0000 7f0710ff9640 3 client.4519.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.251+0000 7f0710ff9640 3 client.4519.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.251+0000 7f0710ff9640 20 client.4519.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.291 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T21:37:40.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.291+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcc88003890 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.292+0000 7fcc77fff640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 0x7fcc6c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.292+0000 7fcc9affd640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 0x7fcc6c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.292+0000 7fcc9affd640 10 monclient: get_auth_request con 0x7fcc6c078e50 auth_method 0 2026-02-22T21:37:40.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.292+0000 7fcc77fff640 10 client.4542.objecter ms_handle_connect 0x7fcc6c078e50 2026-02-22T21:37:40.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.292+0000 7fcc9affd640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 0x7fcc6c07b310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c006fd0 tx=0x7fcc8c006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4044070 con 0x7f65cc1526a0 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 10 monclient: dump: 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 5 monclient: authenticate success, global_id 4545 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 10 monclient: _renew_subs 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65cc14ae90 con 0x7f65cc1526a0 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 10 monclient: _renew_subs 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65cc14b450 con 0x7f65cc1526a0 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.295+0000 7f65d2ec9640 1 librados: init done 2026-02-22T21:37:40.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.296+0000 7f65d2ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.296+0000 7f65d2ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.296+0000 7f65d2ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.296+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65cc160600 con 0x7f65cc1526a0 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fcc8805d3e0 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 10 client.4542.objecter ms_dispatch2 0x7fcc9c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 3 client.4542.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 3 client.4542.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 20 client.4542.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fcc9c06d790 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.301+0000 7fcc77fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b402fbb0 con 0x7f65cc1526a0 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b404e060 con 0x7f65cc1526a0 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c17fa640 10 monclient: dump: 2026-02-22T21:37:40.305 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.304+0000 7f65c2ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:40.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.308+0000 7efd69b03640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:40.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.308+0000 7efd69b03640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:40.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.308+0000 7efd69b03640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.308+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7efd64155db0 con 0x7efd48003dd0 2026-02-22T21:37:40.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.312+0000 7f718ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:40.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.312+0000 7f718ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:40.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.312+0000 7f718ffff640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.314 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.312+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f71901563f0 con 0x7f71900c91f0 2026-02-22T21:37:40.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 10 monclient: monmap: 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.322844+0000 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.322844+0000 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.324 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.325 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 1 Processor -- start 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 1 -- start start 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.322+0000 7f9315cfc640 10 monclient: init 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient: _add_conns ranks=[0,1,4,3,5,2] 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9310155d80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f93100743d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9310156160 0x7f9310070d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-b con 0x7f9310156160 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9310072af0 con 0x7f9310155d80 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9310072dd0 con 0x7f93100743d0 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f93100728a0 con 0x7f9310156160 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.323+0000 7f9315cfc640 10 monclient(hunting): authenticate will time out at 1135.903170s 2026-02-22T21:37:40.344 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2026-02-22T21:37:40.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.346+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd748004070 con 0x7fd74c152ab0 2026-02-22T21:37:40.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.346+0000 7fd732ffd640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 0x7fd71c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.346+0000 7fd752558640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 0x7fd71c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.346+0000 7fd752558640 10 monclient: get_auth_request con 0x7fd71c078e00 auth_method 0 2026-02-22T21:37:40.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.347+0000 7fd732ffd640 10 client.4499.objecter ms_handle_connect 0x7fd71c078e00 2026-02-22T21:37:40.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.347+0000 7fd752558640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 0x7fd71c07b2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009750 tx=0x7fd73c006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.347+0000 7f930ffff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9310156160 0x7f9310070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.348+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fd748012de0 con 0x7fd74c152ab0 2026-02-22T21:37:40.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.348+0000 7fd732ffd640 10 client.4499.objecter ms_dispatch2 0x7fd74c000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.348+0000 7fd732ffd640 3 client.4499.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.348+0000 7fd732ffd640 3 client.4499.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.348+0000 7fd732ffd640 20 client.4499.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9310156160 0x7f9310070d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35766/0 (socket says 10.20.193.51:35766) 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 1 -- 10.20.193.51:0/2031326855 learned_addr learned my addr 10.20.193.51:0/2031326855 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 10 monclient(hunting): get_auth_request con 0x7f9310156160 auth_method 0 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.353+0000 7f930ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0704062430 con 0x7f07140bfc90 2026-02-22T21:37:40.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f0710ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f0710ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd748062270 con 0x7fd74c152ab0 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7fd732ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.355+0000 7fd732ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7f728ce6d640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7f728ce6d640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7f728ce6d640 10 monclient: monmap: 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.359198+0000 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.359198+0000 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.360 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7efd3ffff640 1 -- 10.20.193.51:0/852436486 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd500861c0 con 0x7efd48003dd0 2026-02-22T21:37:40.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7efd3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:40.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.358+0000 7efd3ffff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.361 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:40.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 Processor -- start 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 -- start start 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient: init 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient: _add_conns ranks=[2,1,3,5,4,0] 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7288152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f728806d9d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f72880c95a0 0x7f72881632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-a con 0x7f72880c95a0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f728806f1d0 con 0x7f728806d9d0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f728806de50 con 0x7f7288152ab0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f728806ed40 con 0x7f72880c95a0 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.363 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f728ce6d640 10 monclient(hunting): authenticate will time out at 1135.939628s 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f7286575640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f7288152ab0 0x7f72880c5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33216/0 (socket says 10.20.193.51:33216) 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.359+0000 7f7286575640 1 -- 10.20.193.51:0/3461028747 learned_addr learned my addr 10.20.193.51:0/3461028747 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 10 monclient(hunting): handle_auth_done global_id 4522 payload 274 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 1 -- 10.20.193.51:0/2031326855 >> v1:10.20.193.68:6789/0 conn(0x7f93100743d0 legacy=0x7f931006e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 1 -- 10.20.193.51:0/2031326855 >> v1:10.20.193.51:6789/0 conn(0x7f9310155d80 legacy=0x7f9310073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.360+0000 7f930ffff640 1 -- 10.20.193.51:0/2031326855 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93100730b0 con 0x7f9310156160 2026-02-22T21:37:40.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 10 librados: watch_flush enter 2026-02-22T21:37:40.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 10 librados: watch_flush exit 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 20 client.4487.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 10 client.4487.objecter successfully canceled tick 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 msgr2=0x7efd3407b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 0x7efd3407b2f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd54002770 tx=0x7efd54009040 comp rx=0 tx=0).stop 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 10 monclient: shutdown 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 msgr2=0x7efd641f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.366 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd641f5670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efd50033000 tx=0x7efd640bbb80 comp rx=0 tx=0).stop 2026-02-22T21:37:40.366 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2026-02-22T21:37:40.367 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722561497 0 0) 0x7f728806de50 con 0x7f7288152ab0 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 10 monclient(hunting): my global_id is 4502 2026-02-22T21:37:40.367 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7274003620 con 0x7f7288152ab0 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 shutdown_connections 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efd34078e30 0x7efd3407b2f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd641e7ff0 0x7efd64149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd6414ad00 0x7efd641e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 --2- 10.20.193.51:0/852436486 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd48003dd0 0x7efd641f5670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 >> 10.20.193.51:0/852436486 conn(0x7efd640ce350 msgr2=0x7efd64150200 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 shutdown_connections 2026-02-22T21:37:40.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 -- 10.20.193.51:0/852436486 wait complete. 2026-02-22T21:37:40.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.361+0000 7efd69b03640 1 librados: shutdown 2026-02-22T21:37:40.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.362+0000 7f930ffff640 1 --2- 10.20.193.51:0/2031326855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310070d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9304009e80 tx=0x7f930402f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22aa83ba46b2e89f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2011234247 0 0) 0x7f7274003620 con 0x7f7288152ab0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 >> v2:10.20.193.51:3300/0 conn(0x7f72880c95a0 msgr2=0x7f72881632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 --2- 10.20.193.51:0/3461028747 >> v2:10.20.193.51:3300/0 conn(0x7f72880c95a0 0x7f72881632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 >> v1:10.20.193.68:6789/0 conn(0x7f728806d9d0 legacy=0x7f72880c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728806e130 con 0x7f7288152ab0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f727c002f00 con 0x7f7288152ab0 2026-02-22T21:37:40.371 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: dump: 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 >> v1:10.20.193.125:6789/0 conn(0x7f7288152ab0 legacy=0x7f72880c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 0x7f7274003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): picked mon.c con 0x7f7274003ba0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274004900 0x7f7274004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): picked mon.b con 0x7f7274004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72740054c0 0x7f727400dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): picked mon.a con 0x7f72740054c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f728806ed40 con 0x7f72740054c0 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f727400e670 con 0x7f7274004900 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f727400e870 con 0x7f7274003ba0 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.363+0000 7f7285573640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.364+0000 7f7286575640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 0x7f7274003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): get_auth_request con 0x7f7274003ba0 auth_method 0 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.365+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 10 monclient(hunting): handle_auth_done global_id 4502 payload 293 2026-02-22T21:37:40.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 -- 10.20.193.51:0/3461028747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274004900 msgr2=0x7f7274004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274004900 0x7f7274004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 -- 10.20.193.51:0/3461028747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72740054c0 msgr2=0x7f727400dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72740054c0 0x7f727400dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 monclient: found mon.c 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 1 -- 10.20.193.51:0/3461028747 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274003620 con 0x7f7274003ba0 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286575640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f7286d76640 1 -- 10.20.193.51:0/3461028747 reap_dead start 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f728ce6d640 5 monclient: authenticate success, global_id 4502 2026-02-22T21:37:40.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f728ce6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f716f7fe640 1 -- 10.20.193.51:0/2039482674 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7184061fc0 con 0x7f71900c91f0 2026-02-22T21:37:40.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f716f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:40.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.366+0000 7f716f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.375 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:40.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 1 -- 10.20.193.51:0/2031326855 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930402fdd0 con 0x7f9310156160 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: dump: 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 1 -- 10.20.193.51:0/2031326855 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93040373f0 con 0x7f9310156160 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 5 monclient: authenticate success, global_id 4522 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 1 -- 10.20.193.51:0/2031326855 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93040377c0 con 0x7f9310156160 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 4 set_mon_vals no callback set 2026-02-22T21:37:40.377 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f7286575640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 0x7f7274003fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f727c00c320 tx=0x7f727c02f680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a402ee1ce9ebb3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.378 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2026-02-22T21:37:40.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f930e7fc640 10 monclient: dump: 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 10 monclient: shutdown 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 -- 10.20.193.51:0/2031326855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 msgr2=0x7f9310070d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2031326855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310070d90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9304009e80 tx=0x7f930402f360 comp rx=0 tx=0).stop 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 -- 10.20.193.51:0/2031326855 shutdown_connections 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2031326855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310070d90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 -- 10.20.193.51:0/2031326855 >> 10.20.193.51:0/2031326855 conn(0x7f93100ca250 msgr2=0x7f93100ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 -- 10.20.193.51:0/2031326855 shutdown_connections 2026-02-22T21:37:40.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.367+0000 7f9315cfc640 1 -- 10.20.193.51:0/2031326855 wait complete. 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 10 monclient: monmap: 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 librados: starting msgr at 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 librados: starting objecter 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 Processor -- start 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 -- start start 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 librados: setting wanted keys 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 1 librados: calling monclient init 2026-02-22T21:37:40.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f9315cfc640 10 monclient: init 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 10 librados: watch_flush enter 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 10 librados: watch_flush exit 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 20 client.4510.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 10 client.4510.objecter successfully canceled tick 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 msgr2=0x7f715c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 0x7f715c07b2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f719006e160 tx=0x7f7178009040 comp rx=0 tx=0).stop 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 10 monclient: shutdown 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 msgr2=0x7f71901f7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f71901f7a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f718400c4d0 tx=0x7f718400c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 shutdown_connections 2026-02-22T21:37:40.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f715c078e00 0x7f715c07b2c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7190156010 0x7f71900c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71900caca0 0x7f7190149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 --2- 10.20.193.51:0/2039482674 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71900c91f0 0x7f71901f7a50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 >> 10.20.193.51:0/2039482674 conn(0x7f71900ce340 msgr2=0x7f719015f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 shutdown_connections 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c007e60 con 0x7f7274003ba0 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 -- 10.20.193.51:0/2039482674 wait complete. 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: dump: 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727c005bd0 con 0x7f7274003ba0 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 1 -- 10.20.193.51:0/3461028747 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c007070 con 0x7f7274003ba0 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f728ce6d640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f728ce6d640 4 set_mon_vals no callback set 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f7285573640 10 monclient: dump: 2026-02-22T21:37:40.383 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.368+0000 7f718ffff640 1 librados: shutdown 2026-02-22T21:37:40.384 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f93100743d0 0x7f93101f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-c con 0x7f93100743d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9310155d80 0x7f9310158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-a con 0x7f9310155d80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): picked mon.noname-b con 0x7f9310156160 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9310073670 con 0x7f9310155d80 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f93100712d0 con 0x7f9310156160 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f93100c8150 con 0x7f93100743d0 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f9315cfc640 10 monclient(hunting): authenticate will time out at 1135.949155s 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 10 monclient: shutdown 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 -- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 msgr2=0x7f7274003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 0x7f7274003fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f727c00c320 tx=0x7f727c02f680 comp rx=0 tx=0).stop 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 -- 10.20.193.51:0/3461028747 shutdown_connections 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 --2- 10.20.193.51:0/3461028747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7274003ba0 0x7f7274003fc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 -- 10.20.193.51:0/3461028747 >> 10.20.193.51:0/3461028747 conn(0x7f72880cd7b0 msgr2=0x7f72880cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 -- 10.20.193.51:0/3461028747 shutdown_connections 2026-02-22T21:37:40.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 1 -- 10.20.193.51:0/3461028747 wait complete. 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35784/0 (socket says 10.20.193.51:35784) 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 1 -- 10.20.193.51:0/2624368064 learned_addr learned my addr 10.20.193.51:0/2624368064 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): get_auth_request con 0x7f9310156160 auth_method 0 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930f7fe640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f93100743d0 0x7f93101f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f930ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): get_auth_request con 0x7f93100743d0 auth_method 0 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f930f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.369+0000 7f728ce6d640 10 monclient: monmap: 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.389 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 librados: starting msgr at 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 librados: starting objecter 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 Processor -- start 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 -- start start 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 librados: setting wanted keys 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 librados: calling monclient init 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient: init 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274003ba0 0x7f7288153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-b con 0x7f7274003ba0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-c con 0x7f7288155260 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72881556c0 0x7f72881efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): picked mon.noname-a con 0x7f72881556c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f728806e9d0 con 0x7f72881556c0 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f728806e6f0 con 0x7f7274003ba0 2026-02-22T21:37:40.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7288152e90 con 0x7f7288155260 2026-02-22T21:37:40.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.370+0000 7f728ce6d640 10 monclient(hunting): authenticate will time out at 1135.950660s 2026-02-22T21:37:40.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.8 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 10 monclient(hunting): handle_auth_done global_id 4531 payload 824 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 -- 10.20.193.51:0/2624368064 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f93100743d0 msgr2=0x7f93101f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f93100743d0 0x7f93101f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 -- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9310155d80 msgr2=0x7f9310158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9310155d80 0x7f9310158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.371+0000 7f930ffff640 1 -- 10.20.193.51:0/2624368064 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931014cd50 con 0x7f9310156160 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.372+0000 7f930ffff640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f930402f8c0 tx=0x7f9304004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.372+0000 7f92ebfff640 10 client.?.objecter ms_handle_connect 0x7f9310156160 2026-02-22T21:37:40.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.372+0000 7f92ebfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.393 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2026-02-22T21:37:40.393 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2026-02-22T21:37:40.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.372+0000 7f7286575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274003ba0 0x7f7288153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.372+0000 7f7285d74640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44050/0 (socket says 10.20.193.51:44050) 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 1 -- 10.20.193.51:0/363198135 learned_addr learned my addr 10.20.193.51:0/363198135 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): get_auth_request con 0x7f7288155260 auth_method 0 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7285d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7286575640 10 monclient(hunting): get_auth_request con 0x7f7274003ba0 auth_method 0 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7286575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7286575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.373+0000 7f7286575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.374+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.374+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.374+0000 7f7286575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 -- 10.20.193.51:0/363198135 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274003ba0 msgr2=0x7f7288153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274003ba0 0x7f7288153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 -- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72881556c0 msgr2=0x7f72881efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72881556c0 0x7f72881efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.375+0000 7f7285d74640 1 -- 10.20.193.51:0/363198135 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72881f2b20 con 0x7f7288155260 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f7285d74640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f727000c760 tx=0x7f727000cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f726effd640 10 client.?.objecter ms_handle_connect 0x7f7288155260 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f726effd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304046020 con 0x7f9310156160 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: dump: 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9304037de0 con 0x7f9310156160 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304041690 con 0x7f9310156160 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 5 monclient: authenticate success, global_id 4531 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f930dffb640 4 set_mon_vals no callback set 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 10 monclient: _renew_subs 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931014d030 con 0x7f9310156160 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f92ebfff640 10 monclient: dump: 2026-02-22T21:37:40.397 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 10 monclient: _renew_subs 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f931014d5f0 con 0x7f9310156160 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.376+0000 7f9315cfc640 1 librados: init done 2026-02-22T21:37:40.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72700052d0 con 0x7f7288155260 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: dump: 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 5 monclient: authenticate success, global_id 4508 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _renew_subs 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7288155c90 con 0x7f7288155260 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _renew_subs 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72881f2de0 con 0x7f7288155260 2026-02-22T21:37:40.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 1 librados: init done 2026-02-22T21:37:40.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f9315cfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f9315cfc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f9315cfc640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93100c9c20 con 0x7f9310156160 2026-02-22T21:37:40.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.6 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72880c5ab0 con 0x7f7288155260 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7270002ab0 con 0x7f7288155260 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72700036a0 con 0x7f7288155260 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f7284d72640 4 set_mon_vals no callback set 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.377+0000 7f726effd640 10 monclient: dump: 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.401 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.391+0000 7fcca1e4c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:40.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.391+0000 7fcca1e4c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:40.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.391+0000 7fcca1e4c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.391+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcc9c156380 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.411+0000 7f16dceda640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.411+0000 7f16dceda640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.411+0000 7f16dceda640 10 monclient: monmap: 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.412672+0000 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.412672+0000 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.412+0000 7f16dceda640 1 Processor -- start 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.412+0000 7f16dceda640 1 -- start start 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.412+0000 7f16dceda640 10 monclient: init 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient: _add_conns ranks=[1,4,2,0,5,3] 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f16d8152ab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f16d806d9d0 0x7f16d80c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-b con 0x7f16d806d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16d80c93d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d806f1d0 con 0x7f16d8152ab0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d806de50 con 0x7f16d80c93d0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f16d806ed40 con 0x7f16d806d9d0 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.413+0000 7f16dceda640 10 monclient(hunting): authenticate will time out at 1135.993290s 2026-02-22T21:37:40.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.424+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f65b403f050 con 0x7f65cc1526a0 2026-02-22T21:37:40.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.424+0000 7f65c17fa640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 0x7f659c07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.425+0000 7f65d0c3e640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 0x7f659c07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.425+0000 7f65d0c3e640 10 monclient: get_auth_request con 0x7f659c078e30 auth_method 0 2026-02-22T21:37:40.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.425+0000 7f65d0c3e640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 0x7f659c07b2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009750 tx=0x7f65bc006ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.425+0000 7f65c17fa640 10 client.4545.objecter ms_handle_connect 0x7f659c078e30 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f65b4083060 con 0x7f65cc1526a0 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 10 client.4545.objecter ms_dispatch2 0x7f65cc000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 3 client.4545.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 3 client.4545.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 20 client.4545.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f65cc160600 con 0x7f65cc1526a0 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.437+0000 7f65c17fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.438+0000 7feeae5e4640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.438+0000 7feeae5e4640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.438+0000 7feeae5e4640 10 monclient: monmap: 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.439659+0000 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.439659+0000 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.439+0000 7feeae5e4640 1 Processor -- start 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.439+0000 7feeae5e4640 1 -- start start 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.439+0000 7feeae5e4640 10 monclient: init 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 10 monclient: monmap: 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.440760+0000 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.440760+0000 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.441 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.442 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient: _add_conns ranks=[2,4,0,5,1,3] 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feea8155e60 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7feea815c790 0x7feea80c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-b con 0x7feea815c790 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feea80c92a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea80c9710 con 0x7feea80c92a0 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea80cbc80 con 0x7feea8155e60 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7feea80cc200 con 0x7feea815c790 2026-02-22T21:37:40.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7feeae5e4640 10 monclient(hunting): authenticate will time out at 1136.020218s 2026-02-22T21:37:40.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7fcc77fff640 1 -- 10.20.193.51:0/3885846711 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc88021020 con 0x7fcc9c06d9d0 2026-02-22T21:37:40.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7fcc77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:40.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7fcc77fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.444 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 1 Processor -- start 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 1 -- start start 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.440+0000 7ff5c07d3640 10 monclient: init 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient: _add_conns ranks=[2,0,3,1,5,4] 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5b8155e60 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5b815c790 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff5b80c9470 0x7ff5b80d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b80c9470 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b80cbc80 con 0x7ff5b815c790 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b80cc450 con 0x7ff5b8155e60 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff5b80cc200 con 0x7ff5b80c9470 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.441+0000 7ff5c07d3640 10 monclient(hunting): authenticate will time out at 1136.021240s 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 10 librados: watch_flush enter 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 10 librados: watch_flush exit 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 20 client.4542.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 10 client.4542.objecter successfully canceled tick 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 msgr2=0x7fcc6c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 0x7fcc6c07b310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c006fd0 tx=0x7fcc8c006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 10 monclient: shutdown 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 msgr2=0x7fcc9c1f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800c480 tx=0x7fcc8800c950 comp rx=0 tx=0).stop 2026-02-22T21:37:40.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 shutdown_connections 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc6c078e50 0x7fcc6c07b310 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc9c152ab0 0x7fcc9c1487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc9c0c95a0 0x7fcc9c15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 --2- 10.20.193.51:0/3885846711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc9c06d9d0 0x7fcc9c1f2c90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 >> 10.20.193.51:0/3885846711 conn(0x7fcc9c0cd7b0 msgr2=0x7fcc9c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 shutdown_connections 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 -- 10.20.193.51:0/3885846711 wait complete. 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.442+0000 7fcca1e4c640 1 librados: shutdown 2026-02-22T21:37:40.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.445+0000 7ff5bed49640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff5b80c9470 0x7ff5b80d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff5b80c9470 0x7ff5b80d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33060/0 (socket says 10.20.193.51:33060) 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 1 -- 10.20.193.51:0/1153954124 learned_addr learned my addr 10.20.193.51:0/1153954124 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): get_auth_request con 0x7ff5b80c9470 auth_method 0 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7f071a885640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:40.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7f071a885640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:40.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7f071a885640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f07140c4270 con 0x7f07140bfc90 2026-02-22T21:37:40.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7feeacb5a640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7feea80c92a0 0x7feea80d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47040/0 (socket says 10.20.193.51:47040) 2026-02-22T21:37:40.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7feeacb5a640 1 -- 10.20.193.51:0/2814817369 learned_addr learned my addr 10.20.193.51:0/2814817369 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.446+0000 7ff5bed49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 1 -- 10.20.193.51:0/1153954124 >> v1:10.20.193.125:6789/0 conn(0x7ff5b8155e60 legacy=0x7ff5b80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 1 -- 10.20.193.51:0/1153954124 >> v1:10.20.193.51:6789/0 conn(0x7ff5b815c790 legacy=0x7ff5b80c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.448+0000 7ff5bed49640 1 -- 10.20.193.51:0/1153954124 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b80cc730 con 0x7ff5b80c9470 2026-02-22T21:37:40.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.449+0000 7fd7547e3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:40.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.449+0000 7fd7547e3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:40.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.449+0000 7fd7547e3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.449+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd74c0c8f20 con 0x7fd74c152ab0 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244968410 0 0) 0x7feea80c9710 con 0x7feea80c92a0 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 10 monclient(hunting): my global_id is 4551 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.450+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee8c003620 con 0x7feea80c92a0 2026-02-22T21:37:40.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.451+0000 7ff5bed49640 1 --2- 10.20.193.51:0/1153954124 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b80c9470 0x7ff5b80d1810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009e50 tx=0x7ff5b402f2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326a0706543f1264 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.0 2026-02-22T21:37:40.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129240978 0 0) 0x7fee8c003620 con 0x7feea80c92a0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 >> v2:10.20.193.68:3300/0 conn(0x7feea815c790 msgr2=0x7feea80c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 --2- 10.20.193.51:0/2814817369 >> v2:10.20.193.68:3300/0 conn(0x7feea815c790 0x7feea80c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 >> v1:10.20.193.125:6789/0 conn(0x7feea8155e60 legacy=0x7feea80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea80cc450 con 0x7feea80c92a0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fee9c003280 con 0x7feea80c92a0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: dump: 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.456 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 >> v1:10.20.193.51:6789/0 conn(0x7feea80c92a0 legacy=0x7feea80d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fee8c003ba0 0x7fee8c003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): picked mon.c con 0x7fee8c003ba0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7fee8c004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): picked mon.a con 0x7fee8c004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fee8c005510 0x7fee8c00bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): picked mon.b con 0x7fee8c005510 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7feea80cc200 con 0x7fee8c004900 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c00c450 con 0x7fee8c005510 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c00c650 con 0x7fee8c003ba0 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.453+0000 7feea6ffd640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 1 -- 10.20.193.51:0/1153954124 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b402fd00 con 0x7ff5b80c9470 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: dump: 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 1 -- 10.20.193.51:0/1153954124 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5b40373f0 con 0x7ff5b80c9470 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 1 -- 10.20.193.51:0/1153954124 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b40377c0 con 0x7ff5b80c9470 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5bd546640 10 monclient: dump: 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5c07d3640 5 monclient: authenticate success, global_id 4548 2026-02-22T21:37:40.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5c07d3640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7ff5c07d3640 4 set_mon_vals no callback set 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.459+0000 7feea77fe640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7fee8c004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 10 monclient: shutdown 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 -- 10.20.193.51:0/1153954124 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b80c9470 msgr2=0x7ff5b80d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/1153954124 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b80c9470 0x7ff5b80d1810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009e50 tx=0x7ff5b402f2a0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): get_auth_request con 0x7fee8c004900 auth_method 0 2026-02-22T21:37:40.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 -- 10.20.193.51:0/1153954124 shutdown_connections 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/1153954124 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b80c9470 0x7ff5b80d1810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 -- 10.20.193.51:0/1153954124 >> 10.20.193.51:0/1153954124 conn(0x7ff5b806d8c0 msgr2=0x7ff5b806dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 -- 10.20.193.51:0/1153954124 shutdown_connections 2026-02-22T21:37:40.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7ff5c07d3640 1 -- 10.20.193.51:0/1153954124 wait complete. 2026-02-22T21:37:40.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.460+0000 7feea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:40.466 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2026-02-22T21:37:40.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 10 monclient: monmap: 2026-02-22T21:37:40.466 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.466 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 librados: starting msgr at 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 librados: starting objecter 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 Processor -- start 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 -- start start 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 librados: setting wanted keys 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 1 librados: calling monclient init 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.461+0000 7ff5c07d3640 10 monclient: init 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:40.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff5b80c9470 0x7ff5b81499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b80c9470 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b8155e60 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff5b815c790 0x7ff5b81f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b815c790 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b80ccfd0 con 0x7ff5b8155e60 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b80cccf0 con 0x7ff5b80c9470 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b80cd2b0 con 0x7ff5b815c790 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7ff5c07d3640 10 monclient(hunting): authenticate will time out at 1136.042090s 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 10 monclient(hunting): handle_auth_done global_id 4551 payload 293 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 -- 10.20.193.51:0/2814817369 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fee8c003ba0 msgr2=0x7fee8c003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fee8c003ba0 0x7fee8c003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 -- 10.20.193.51:0/2814817369 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fee8c005510 msgr2=0x7fee8c00bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fee8c005510 0x7fee8c00bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 monclient: found mon.a 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feeacb5a640 1 -- 10.20.193.51:0/2814817369 reap_dead start 2026-02-22T21:37:40.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 1 -- 10.20.193.51:0/2814817369 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c003620 con 0x7fee8c004900 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feeae5e4640 5 monclient: authenticate success, global_id 4551 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.462+0000 7feeae5e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.464+0000 7feea77fe640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7fee8c004d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fee98009980 tx=0x7fee9802ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944523853d2bf8b9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9803c070 con 0x7fee8c004900 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: dump: 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9802f7b0 con 0x7fee8c004900 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 1 -- 10.20.193.51:0/2814817369 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98009550 con 0x7fee8c004900 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feea6ffd640 10 monclient: dump: 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feeae5e4640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7feeae5e4640 4 set_mon_vals no callback set 2026-02-22T21:37:40.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33092/0 (socket says 10.20.193.51:33092) 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 1 -- 10.20.193.51:0/326625139 learned_addr learned my addr 10.20.193.51:0/326625139 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 10 monclient(hunting): get_auth_request con 0x7ff5b8155e60 auth_method 0 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.466+0000 7ff5bdd47640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7ff5bdd47640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7ff5bdd47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7ff5bdd47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 10 monclient: shutdown 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 -- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 msgr2=0x7fee8c004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7fee8c004d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fee98009980 tx=0x7fee9802ed50 comp rx=0 tx=0).stop 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 -- 10.20.193.51:0/2814817369 shutdown_connections 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2814817369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7fee8c004d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 -- 10.20.193.51:0/2814817369 >> 10.20.193.51:0/2814817369 conn(0x7feea806d8c0 msgr2=0x7feea806dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 -- 10.20.193.51:0/2814817369 shutdown_connections 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.467+0000 7feeae5e4640 1 -- 10.20.193.51:0/2814817369 wait complete. 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 10 monclient: monmap: 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.474 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 librados: starting msgr at 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 librados: starting objecter 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 Processor -- start 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 -- start start 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 librados: setting wanted keys 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 1 librados: calling monclient init 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7feeae5e4640 10 monclient: init 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 10 monclient(hunting): handle_auth_done global_id 4554 payload 824 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 -- 10.20.193.51:0/326625139 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff5b815c790 msgr2=0x7ff5b81f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff5b815c790 0x7ff5b81f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 -- 10.20.193.51:0/326625139 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff5b80c9470 msgr2=0x7ff5b81499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff5b80c9470 0x7ff5b81499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.468+0000 7ff5bdd47640 1 -- 10.20.193.51:0/326625139 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b8149ee0 con 0x7ff5b8155e60 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-a con 0x7fee8c004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feea8143b20 0x7feea815d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-c con 0x7feea8143b20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feea815d640 0x7feea815a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): picked mon.noname-b con 0x7feea815d640 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7feea80cccf0 con 0x7fee8c004900 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7feea80cca10 con 0x7feea815d640 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7feea80ccfd0 con 0x7feea8143b20 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.469+0000 7feeae5e4640 10 monclient(hunting): authenticate will time out at 1136.049118s 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 10 monclient: monmap: 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.470844+0000 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.470844+0000 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.478 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 1 Processor -- start 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 1 -- start start 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.470+0000 7f91c4311640 10 monclient: init 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7ff5bdd47640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00b500 tx=0x7ff5ac00b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7ff59effd640 10 client.?.objecter ms_handle_connect 0x7ff5b8155e60 2026-02-22T21:37:40.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7ff59effd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient: _add_conns ranks=[1,0,2,4,3,5] 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f91bc1526a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91bc152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91bc15d3b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91bc073ae0 con 0x7f91bc152a80 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91bc073d60 con 0x7f91bc1526a0 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91bc074040 con 0x7f91bc15d3b0 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7f91c4311640 10 monclient(hunting): authenticate will time out at 1136.051317s 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33108/0 (socket says 10.20.193.51:33108) 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 1 -- 10.20.193.51:0/2149629945 learned_addr learned my addr 10.20.193.51:0/2149629945 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 10 monclient(hunting): get_auth_request con 0x7fee8c004900 auth_method 0 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.471+0000 7feea7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.472+0000 7feea7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.472+0000 7feea7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.472+0000 7feea7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 10 monclient(hunting): handle_auth_done global_id 4557 payload 824 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 -- 10.20.193.51:0/2149629945 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feea8143b20 msgr2=0x7feea815d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feea8143b20 0x7feea815d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 -- 10.20.193.51:0/2149629945 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feea815d640 msgr2=0x7feea815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feea815d640 0x7feea815a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 monclient: found mon.noname-a 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.475+0000 7feea7fff640 1 -- 10.20.193.51:0/2149629945 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea815f970 con 0x7fee8c004900 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac004480 con 0x7ff5b8155e60 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: dump: 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.482 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ac014070 con 0x7ff5b8155e60 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 5 monclient: authenticate success, global_id 4554 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 10 monclient: _renew_subs 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b81f3130 con 0x7ff5b8155e60 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac002920 con 0x7ff5b8155e60 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5bcd45640 4 set_mon_vals no callback set 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff59effd640 10 monclient: dump: 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 10 monclient: _renew_subs 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b81f40d0 con 0x7ff5b8155e60 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.476+0000 7ff5c07d3640 1 librados: init done 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7ff5c07d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7ff5c07d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7ff5c07d3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b815cbd0 con 0x7ff5b8155e60 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7feea7fff640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fee9400b450 tx=0x7fee9400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7feea4ff9640 10 client.?.objecter ms_handle_connect 0x7fee8c004900 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.477+0000 7feea4ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 10 monclient: monmap: 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.484 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.481097+0000 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.481097+0000 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 1 Processor -- start 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 1 -- start start 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.480+0000 7f8a8d474640 10 monclient: init 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9304041830 con 0x7f9310156160 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient: _add_conns ranks=[2,0,4,5,3,1] 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a88152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a8806d9d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8a880c95a0 0x7f8a881632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-b con 0x7f8a880c95a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a8806f1d0 con 0x7f8a8806d9d0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a8806de50 con 0x7f8a88152ab0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f8a8806ed40 con 0x7f8a880c95a0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f8a8d474640 10 monclient(hunting): authenticate will time out at 1136.061691s 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f92ebfff640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 0x7f92e007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f903b67c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.481+0000 7f903b67c640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f903b67c640 10 monclient: monmap: 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.482677+0000 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.482677+0000 2026-02-22T21:37:40.486 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f930f7fe640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 0x7f92e007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f930f7fe640 10 monclient: get_auth_request con 0x7f92e0078e50 auth_method 0 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f92ebfff640 10 client.4531.objecter ms_handle_connect 0x7f92e0078e50 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f930f7fe640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 0x7f92e007b310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92fc004660 tx=0x7f92fc009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f903b67c640 1 Processor -- start 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f903b67c640 1 -- start start 2026-02-22T21:37:40.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f903b67c640 10 monclient: init 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f391c61a640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f391c61a640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.482+0000 7f391c61a640 10 monclient: monmap: 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.483213+0000 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.483213+0000 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 Processor -- start 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient: _add_conns ranks=[0,4,3,2,1,5] 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 -- start start 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient: init 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90341526a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9034152a80 0x7f90340cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-b con 0x7f9034152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f90340cdb20 0x7f903415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-a con 0x7f90340cdb20 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9034073ae0 con 0x7f90341526a0 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f90340cca00 con 0x7f90340cdb20 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9034074c50 con 0x7f9034152a80 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f903b67c640 10 monclient(hunting): authenticate will time out at 1136.063172s 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f9304084120 con 0x7f9310156160 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f92ebfff640 10 client.4531.objecter ms_dispatch2 0x7f9310000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f92ebfff640 3 client.4531.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f92ebfff640 3 client.4531.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient: _add_conns ranks=[4,0,3,1,2,5] 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3914155e60 0x7f391415c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-b con 0x7f3914155e60 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f92ebfff640 20 client.4531.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39140cb570 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f39140c9470 0x7f39140d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-a con 0x7f39140c9470 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39140cc450 con 0x7f39140cb570 2026-02-22T21:37:40.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f391415ccd0 con 0x7f39140c9470 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f39140cc200 con 0x7f3914155e60 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.483+0000 7f391c61a640 10 monclient(hunting): authenticate will time out at 1136.063662s 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.485+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7270003840 con 0x7f7288155260 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.486+0000 7f726effd640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 0x7f725407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.486+0000 7f7286575640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 0x7f725407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.486+0000 7f7286575640 10 monclient: get_auth_request con 0x7f7254078e00 auth_method 0 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.486+0000 7f7286575640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 0x7f725407b2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f727c036000 tx=0x7f727c006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.486+0000 7f726effd640 10 client.4508.objecter ms_handle_connect 0x7f7254078e00 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.487+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f7270097a40 con 0x7f7288155260 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.487+0000 7f726effd640 10 client.4508.objecter ms_dispatch2 0x7f7288000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.487+0000 7f726effd640 3 client.4508.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.487+0000 7f726effd640 3 client.4508.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.487+0000 7f726effd640 20 client.4508.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.490+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f93100c9c20 con 0x7f9310156160 2026-02-22T21:37:40.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.490+0000 7f92ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.490+0000 7f92ebfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f16d806d9d0 0x7f16d80c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f16d806d9d0 0x7f16d80c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35800/0 (socket says 10.20.193.51:35800) 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 1 -- 10.20.193.51:0/1887373172 learned_addr learned my addr 10.20.193.51:0/1887373172 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 10 monclient(hunting): get_auth_request con 0x7f16d806d9d0 auth_method 0 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.492+0000 7f16d5d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.493+0000 7f16d5d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.493+0000 7f16d5d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.493+0000 7f16d5d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7270062750 con 0x7f7288155260 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f726effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f726effd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 10 monclient(hunting): handle_auth_done global_id 4537 payload 274 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 1 -- 10.20.193.51:0/1887373172 >> v1:10.20.193.125:6789/0 conn(0x7f16d80c93d0 legacy=0x7f16d8158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 1 -- 10.20.193.51:0/1887373172 >> v1:10.20.193.68:6789/0 conn(0x7f16d8152ab0 legacy=0x7f16d80c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.494+0000 7f16d5d74640 1 -- 10.20.193.51:0/1887373172 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d806e130 con 0x7f16d806d9d0 2026-02-22T21:37:40.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.496+0000 7f16d5d74640 1 --2- 10.20.193.51:0/1887373172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d806d9d0 0x7f16d80c8e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16c00099a0 tx=0x7f16c002efe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc32b4efd941eb9c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.497+0000 7f8a877fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8a880c95a0 0x7f8a881632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.497+0000 7f9038bf0640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9034152a80 0x7f90340cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8a880c95a0 0x7f8a881632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35854/0 (socket says 10.20.193.51:35854) 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 1 -- 10.20.193.51:0/2286687724 learned_addr learned my addr 10.20.193.51:0/2286687724 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): get_auth_request con 0x7f8a880c95a0 auth_method 0 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9034152a80 0x7f90340cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35860/0 (socket says 10.20.193.51:35860) 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 1 -- 10.20.193.51:0/3796899633 learned_addr learned my addr 10.20.193.51:0/3796899633 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): get_auth_request con 0x7f9034152a80 auth_method 0 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f8a877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.498+0000 7f9038bf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 10 monclient(hunting): handle_auth_done global_id 4540 payload 274 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 1 -- 10.20.193.51:0/2286687724 >> v1:10.20.193.125:6789/0 conn(0x7f8a88152ab0 legacy=0x7f8a880c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 1 -- 10.20.193.51:0/2286687724 >> v1:10.20.193.51:6789/0 conn(0x7f8a8806d9d0 legacy=0x7f8a880c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.499+0000 7f8a877fe640 1 -- 10.20.193.51:0/2286687724 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8806e130 con 0x7f8a880c95a0 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 10 monclient(hunting): handle_auth_done global_id 4543 payload 274 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 1 -- 10.20.193.51:0/3796899633 >> v2:10.20.193.51:3300/0 conn(0x7f90340cdb20 msgr2=0x7f903415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 1 --2- 10.20.193.51:0/3796899633 >> v2:10.20.193.51:3300/0 conn(0x7f90340cdb20 0x7f903415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 1 -- 10.20.193.51:0/3796899633 >> v1:10.20.193.51:6789/0 conn(0x7f90341526a0 legacy=0x7f90340733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.500+0000 7f9038bf0640 1 -- 10.20.193.51:0/3796899633 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9034073d60 con 0x7f9034152a80 2026-02-22T21:37:40.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3914155e60 0x7f391415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f91c2887640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f91bc15d3b0 0x7f91bc15f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33248/0 (socket says 10.20.193.51:33248) 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f91c2887640 1 -- 10.20.193.51:0/1312805412 learned_addr learned my addr 10.20.193.51:0/1312805412 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3914155e60 0x7f391415c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35866/0 (socket says 10.20.193.51:35866) 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 1 -- 10.20.193.51:0/2614673371 learned_addr learned my addr 10.20.193.51:0/2614673371 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 10 monclient(hunting): get_auth_request con 0x7f3914155e60 auth_method 0 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.501+0000 7f391a38f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7fd732ffd640 1 -- 10.20.193.51:0/3730088509 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd748061b70 con 0x7fd74c152ab0 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7fd732ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:40.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7fd732ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.502+0000 7f8a877fe640 1 --2- 10.20.193.51:0/2286687724 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a880c95a0 0x7f8a881632e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e80 tx=0x7f8a7802f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f62cdf5a1f5d38d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 10 monclient(hunting): handle_auth_done global_id 4546 payload 274 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 1 -- 10.20.193.51:0/2614673371 >> v2:10.20.193.51:3300/0 conn(0x7f39140c9470 msgr2=0x7f39140d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 1 --2- 10.20.193.51:0/2614673371 >> v2:10.20.193.51:3300/0 conn(0x7f39140c9470 0x7f39140d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 1 -- 10.20.193.51:0/2614673371 >> v1:10.20.193.51:6789/0 conn(0x7f39140cb570 legacy=0x7f39140c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.503+0000 7f391a38f640 1 -- 10.20.193.51:0/2614673371 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39140cc730 con 0x7f3914155e60 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 10 librados: watch_flush enter 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 10 librados: watch_flush exit 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 20 client.4499.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 10 client.4499.objecter successfully canceled tick 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 msgr2=0x7fd71c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 0x7fd71c07b2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009750 tx=0x7fd73c006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 10 monclient: shutdown 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 msgr2=0x7fd74c1f3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd74800b9a0 tx=0x7fd74800be70 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 shutdown_connections 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd71c078e00 0x7fd71c07b2c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd74c152ab0 0x7fd74c1f3aa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd74c0c93d0 0x7fd74c148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 --2- 10.20.193.51:0/3730088509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd74c06d9d0 0x7fd74c1f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 >> 10.20.193.51:0/3730088509 conn(0x7fd74c0cd7b0 msgr2=0x7fd74c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 shutdown_connections 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 -- 10.20.193.51:0/3730088509 wait complete. 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7fd7547e3640 1 librados: shutdown 2026-02-22T21:37:40.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.504+0000 7f9038bf0640 1 --2- 10.20.193.51:0/3796899633 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9034152a80 0x7f90340cd5e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f903415f410 tx=0x7f902802ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a61a18f05e50bcf1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.505+0000 7f391a38f640 1 --2- 10.20.193.51:0/2614673371 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3914155e60 0x7f391415c790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f391408ed10 tx=0x7f390402eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f478110c1b71de5e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 1 -- 10.20.193.51:0/2286687724 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7802fd80 con 0x7f8a880c95a0 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: dump: 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.510 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 1 -- 10.20.193.51:0/2286687724 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a780373f0 con 0x7f8a880c95a0 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 5 monclient: authenticate success, global_id 4540 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 1 -- 10.20.193.51:0/2286687724 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a780377c0 con 0x7f8a880c95a0 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 4 set_mon_vals no callback set 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a85ffb640 10 monclient: dump: 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.511 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 1 -- 10.20.193.51:0/1887373172 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c003c070 con 0x7f16d806d9d0 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: dump: 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 1 -- 10.20.193.51:0/1887373172 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c002fc10 con 0x7f16d806d9d0 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 1 -- 10.20.193.51:0/1887373172 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0037440 con 0x7f16d806d9d0 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 10 monclient: shutdown 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 -- 10.20.193.51:0/2286687724 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a880c95a0 msgr2=0x7f8a881632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 --2- 10.20.193.51:0/2286687724 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a880c95a0 0x7f8a881632e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e80 tx=0x7f8a7802f360 comp rx=0 tx=0).stop 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16dceda640 5 monclient: authenticate success, global_id 4537 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16dceda640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16dceda640 4 set_mon_vals no callback set 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f16d5573640 10 monclient: dump: 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 -- 10.20.193.51:0/2286687724 shutdown_connections 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 --2- 10.20.193.51:0/2286687724 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a880c95a0 0x7f8a881632e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 -- 10.20.193.51:0/2286687724 >> 10.20.193.51:0/2286687724 conn(0x7f8a880cd7b0 msgr2=0x7f8a880cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 -- 10.20.193.51:0/2286687724 shutdown_connections 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.509+0000 7f8a8d474640 1 -- 10.20.193.51:0/2286687724 wait complete. 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient: monmap: 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.513 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 10 monclient: shutdown 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 -- 10.20.193.51:0/1887373172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d806d9d0 msgr2=0x7f16d80c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 --2- 10.20.193.51:0/1887373172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d806d9d0 0x7f16d80c8e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f16c00099a0 tx=0x7f16c002efe0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 -- 10.20.193.51:0/1887373172 shutdown_connections 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 --2- 10.20.193.51:0/1887373172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d806d9d0 0x7f16d80c8e90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 -- 10.20.193.51:0/1887373172 >> 10.20.193.51:0/1887373172 conn(0x7f16d80cd7b0 msgr2=0x7f16d80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 -- 10.20.193.51:0/1887373172 shutdown_connections 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 -- 10.20.193.51:0/1887373172 wait complete. 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 librados: starting msgr at 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 librados: starting objecter 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 Processor -- start 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 -- start start 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 librados: setting wanted keys 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 librados: calling monclient init 2026-02-22T21:37:40.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient: init 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 10 monclient: monmap: 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a8806d9d0 0x7f8a881e8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-b con 0x7f8a8806d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-c con 0x7f8a880c95a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a88152ab0 0x7f8a881f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): picked mon.noname-a con 0x7f8a88152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8806e9d0 con 0x7f8a88152ab0 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8806e6f0 con 0x7f8a8806d9d0 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8a880c61c0 con 0x7f8a880c95a0 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f8a8d474640 10 monclient(hunting): authenticate will time out at 1136.090846s 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 librados: starting msgr at 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.510+0000 7f16dceda640 1 librados: starting objecter 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 Processor -- start 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 -- start start 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 librados: setting wanted keys 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 librados: calling monclient init 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient: init 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44094/0 (socket says 10.20.193.51:44094) 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 1 -- 10.20.193.51:0/350490732 learned_addr learned my addr 10.20.193.51:0/350490732 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): get_auth_request con 0x7f8a880c95a0 auth_method 0 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-c con 0x7f16d806d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16d80c93d0 0x7f16d81487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-a con 0x7f16d80c93d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d8152ab0 0x7f16d81e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): picked mon.noname-b con 0x7f16d8152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f16d806e6f0 con 0x7f16d80c93d0 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80c9880 con 0x7f16d8152ab0 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f16d806e9d0 con 0x7f16d806d9d0 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f16dceda640 10 monclient(hunting): authenticate will time out at 1136.091468s 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f8a867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f391938d640 1 -- 10.20.193.51:0/2614673371 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390403c070 con 0x7f3914155e60 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.511+0000 7f391938d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: dump: 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 1 -- 10.20.193.51:0/2614673371 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f390402fb40 con 0x7f3914155e60 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 5 monclient: authenticate success, global_id 4546 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 4 set_mon_vals no callback set 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 1 -- 10.20.193.51:0/2614673371 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904037440 con 0x7f3914155e60 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391938d640 10 monclient: dump: 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.518 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.519 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.2 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 10 monclient: shutdown 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 -- 10.20.193.51:0/2614673371 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3914155e60 msgr2=0x7f391415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 --2- 10.20.193.51:0/2614673371 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3914155e60 0x7f391415c790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f391408ed10 tx=0x7f390402eed0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 -- 10.20.193.51:0/2614673371 shutdown_connections 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 --2- 10.20.193.51:0/2614673371 >> v2:10.20.193.51:3300/0 conn(0x7f39140c9470 0x7f39140d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 --2- 10.20.193.51:0/2614673371 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3914155e60 0x7f391415c790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 -- 10.20.193.51:0/2614673371 >> 10.20.193.51:0/2614673371 conn(0x7f391406d8c0 msgr2=0x7f391406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 -- 10.20.193.51:0/2614673371 shutdown_connections 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.512+0000 7f391c61a640 1 -- 10.20.193.51:0/2614673371 wait complete. 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient: monmap: 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2026-02-22T21:37:40.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 -- 10.20.193.51:0/350490732 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a8806d9d0 msgr2=0x7f8a881e8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a8806d9d0 0x7f8a881e8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 -- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a88152ab0 msgr2=0x7f8a881f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a88152ab0 0x7f8a881f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f8a867fc640 1 -- 10.20.193.51:0/350490732 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a88073700 con 0x7f8a880c95a0 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 librados: starting msgr at 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 librados: starting objecter 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 Processor -- start 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 -- start start 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 librados: setting wanted keys 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 librados: calling monclient init 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient: init 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39140c9470 0x7f39141f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-b con 0x7f39140c9470 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f39140cb570 0x7f391415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-a con 0x7f39140cb570 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): picked mon.noname-c con 0x7f3914155e60 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f39140ccfd0 con 0x7f39140cb570 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f39140cccf0 con 0x7f39140c9470 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f39140cd2b0 con 0x7f3914155e60 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.513+0000 7f391c61a640 10 monclient(hunting): authenticate will time out at 1136.093765s 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44106/0 (socket says 10.20.193.51:44106) 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 1 -- 10.20.193.51:0/1322161604 learned_addr learned my addr 10.20.193.51:0/1322161604 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 10 monclient(hunting): get_auth_request con 0x7f16d806d9d0 auth_method 0 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f16d6575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f391ab90640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f8a867fc640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b550 tx=0x7f8a7000ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f8a677fe640 10 client.?.objecter ms_handle_connect 0x7f8a880c95a0 2026-02-22T21:37:40.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.514+0000 7f8a677fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 1 -- 10.20.193.51:0/3796899633 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902803c070 con 0x7f9034152a80 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: dump: 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 1 -- 10.20.193.51:0/3796899633 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f902802fbd0 con 0x7f9034152a80 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 1 -- 10.20.193.51:0/3796899633 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028037440 con 0x7f9034152a80 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f9023fff640 10 monclient: dump: 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 5 monclient: authenticate success, global_id 4543 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 4 set_mon_vals no callback set 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44112/0 (socket says 10.20.193.51:44112) 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 1 -- 10.20.193.51:0/1326221703 learned_addr learned my addr 10.20.193.51:0/1326221703 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 10 monclient(hunting): get_auth_request con 0x7f3914155e60 auth_method 0 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f391ab90640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f16d6575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f16d6575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f16d6575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 10 monclient: shutdown 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 -- 10.20.193.51:0/3796899633 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9034152a80 msgr2=0x7f90340cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 --2- 10.20.193.51:0/3796899633 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9034152a80 0x7f90340cd5e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f903415f410 tx=0x7f902802ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 -- 10.20.193.51:0/3796899633 shutdown_connections 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 --2- 10.20.193.51:0/3796899633 >> v2:10.20.193.51:3300/0 conn(0x7f90340cdb20 0x7f903415d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 --2- 10.20.193.51:0/3796899633 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9034152a80 0x7f90340cd5e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 -- 10.20.193.51:0/3796899633 >> 10.20.193.51:0/3796899633 conn(0x7f90340ce350 msgr2=0x7f90340ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f903b67c640 1 -- 10.20.193.51:0/3796899633 shutdown_connections 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.515+0000 7f16d6d76640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d8152ab0 0x7f16d81e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f391ab90640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f391ab90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f391ab90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): get_auth_request con 0x7f16d8152ab0 auth_method 0 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f903b67c640 1 -- 10.20.193.51:0/3796899633 wait complete. 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f16d6d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f903b67c640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f903b67c640 10 monclient: monmap: 2026-02-22T21:37:40.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f391a38f640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39140c9470 0x7f39141f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f0710ff9640 1 -- 10.20.193.51:0/1558587666 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0704066730 con 0x7f07140bfc90 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f0710ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.516+0000 7f0710ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 librados: starting msgr at 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 librados: starting objecter 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 Processor -- start 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 -- start start 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 librados: setting wanted keys 2026-02-22T21:37:40.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 librados: calling monclient init 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient: init 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 10 monclient(hunting): handle_auth_done global_id 4514 payload 824 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 -- 10.20.193.51:0/1322161604 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d8152ab0 msgr2=0x7f16d81e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d8152ab0 0x7f16d81e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 -- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16d80c93d0 msgr2=0x7f16d81487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16d80c93d0 0x7f16d81487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f16d6575640 1 -- 10.20.193.51:0/1322161604 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d815a290 con 0x7f16d806d9d0 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-b con 0x7f90340cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90341526a0 0x7f90340ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-a con 0x7f90341526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9034152a80 0x7f903415dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): picked mon.noname-c con 0x7f9034152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9034074600 con 0x7f90341526a0 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9034074320 con 0x7f90340cdb20 2026-02-22T21:37:40.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f90340748e0 con 0x7f9034152a80 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f903b67c640 10 monclient(hunting): authenticate will time out at 1136.097630s 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70013020 con 0x7f8a880c95a0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 10 monclient: dump: 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 5 monclient: authenticate success, global_id 4511 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 10 monclient: _renew_subs 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a880739e0 con 0x7f8a880c95a0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 10 monclient: _renew_subs 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a88073fa0 con 0x7f8a880c95a0 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.517+0000 7f8a8d474640 1 librados: init done 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f391a38f640 10 monclient(hunting): get_auth_request con 0x7f39140c9470 auth_method 0 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f391a38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f391a38f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f391a38f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f8a8d474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f8a8d474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f8a8d474640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.518+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a88152e90 con 0x7f8a880c95a0 2026-02-22T21:37:40.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 10 librados: watch_flush enter 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 10 librados: watch_flush exit 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 20 client.4519.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 10 client.4519.objecter successfully canceled tick 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 msgr2=0x7f06e407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 0x7f06e407b310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0714156280 tx=0x7f0700031000 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 10 monclient: shutdown 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 msgr2=0x7f07140748d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f070400e8c0 tx=0x7f070400ed90 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 shutdown_connections 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f06e4078e50 0x7f06e407b310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07141527b0 0x7f071414ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f07140bfc90 0x7f07140748d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 --2- 10.20.193.51:0/1558587666 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f07140bf800 0x7f07140c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 >> 10.20.193.51:0/1558587666 conn(0x7f07140cd7b0 msgr2=0x7f071415f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 shutdown_connections 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 -- 10.20.193.51:0/1558587666 wait complete. 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f071a885640 1 librados: shutdown 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391a38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 -- 10.20.193.51:0/1326221703 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39140c9470 msgr2=0x7f39141f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39140c9470 0x7f39141f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 -- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f39140cb570 msgr2=0x7f391415eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f39140cb570 0x7f391415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.519+0000 7f391ab90640 1 -- 10.20.193.51:0/1326221703 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391414dc40 con 0x7f3914155e60 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a70004480 con 0x7f8a880c95a0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a700029a0 con 0x7f8a880c95a0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a857fa640 4 set_mon_vals no callback set 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.520+0000 7f8a677fe640 10 monclient: dump: 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.521+0000 7f391ab90640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f391000b970 tx=0x7f391000be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.521+0000 7f38faffd640 10 client.?.objecter ms_handle_connect 0x7f3914155e60 2026-02-22T21:37:40.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.521+0000 7f38faffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.522+0000 7f16d6575640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b4d0 tx=0x7f16cc00b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.522+0000 7f16beffd640 10 client.?.objecter ms_handle_connect 0x7f16d806d9d0 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.522+0000 7f16beffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.522+0000 7f90393f1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35908/0 (socket says 10.20.193.51:35908) 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 1 -- 10.20.193.51:0/1101116074 learned_addr learned my addr 10.20.193.51:0/1101116074 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): get_auth_request con 0x7f90340cdb20 auth_method 0 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.523+0000 7f90393f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 10 monclient(hunting): handle_auth_done global_id 4555 payload 824 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 -- 10.20.193.51:0/1101116074 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9034152a80 msgr2=0x7f903415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9034152a80 0x7f903415dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 -- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90341526a0 msgr2=0x7f90340ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90341526a0 0x7f90340ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.525+0000 7f90393f1640 1 -- 10.20.193.51:0/1101116074 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90341f4710 con 0x7f90340cdb20 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.526+0000 7f90393f1640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f902400b4d0 tx=0x7f902400b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.526+0000 7f9021ffb640 10 client.?.objecter ms_handle_connect 0x7f90340cdb20 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.526+0000 7f9021ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.3 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f65d2ec9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f65d2ec9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f65d2ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 15} v 0) -- 0x7f65cc0cbf20 con 0x7f65cc1526a0 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024004480 con 0x7f90340cdb20 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: dump: 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9024015070 con 0x7f90340cdb20 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024002920 con 0x7f90340cdb20 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f9021ffb640 10 monclient: dump: 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f90237fe640 4 set_mon_vals no callback set 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 5 monclient: authenticate success, global_id 4555 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 10 monclient: _renew_subs 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903415d120 con 0x7f90340cdb20 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 10 monclient: _renew_subs 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90341f4150 con 0x7f90340cdb20 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.528+0000 7f903b67c640 1 librados: init done 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.529+0000 7f903b67c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.529+0000 7f903b67c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.529+0000 7f903b67c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.529+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90341f4430 con 0x7f90340cdb20 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.539+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9024002ac0 con 0x7f90340cdb20 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.540+0000 7f9021ffb640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 0x7f900407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.540+0000 7f9038bf0640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 0x7f900407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.540+0000 7f9038bf0640 10 monclient: get_auth_request con 0x7f9004078e00 auth_method 0 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.540+0000 7f9021ffb640 10 client.4555.objecter ms_handle_connect 0x7f9004078e00 2026-02-22T21:37:40.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.540+0000 7f9038bf0640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 0x7f900407b2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f903406cf40 tx=0x7f9028031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 10 monclient: get_monmap_and_config 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 10 monclient: monmap: 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:40.541773+0000 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:40.541773+0000 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f902405e2a0 con 0x7f90340cdb20 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7f9021ffb640 10 client.4555.objecter ms_dispatch2 0x7f9034000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7f9021ffb640 3 client.4555.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7f9021ffb640 3 client.4555.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7f9021ffb640 20 client.4555.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 1 Processor -- start 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 1 -- start start 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.541+0000 7efd2e032640 10 monclient: init 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient: _add_conns ranks=[5,0,4,3,1,2] 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7efd281526a0 0x7efd280733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-c con 0x7efd281526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd28152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 0x7efd2815d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-b con 0x7efd2815d3b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd28073d60 con 0x7efd28152a80 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7efd280cca00 con 0x7efd2815d3b0 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7efd28074c50 con 0x7efd281526a0 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.542+0000 7efd2e032640 10 monclient(hunting): authenticate will time out at 1136.122215s 2026-02-22T21:37:40.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.580+0000 7f9315cfc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:40.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.580+0000 7f9315cfc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:40.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.580+0000 7f9315cfc640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.580+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 12} v 0) -- 0x7f93100c7300 con 0x7f9310156160 2026-02-22T21:37:40.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.582+0000 7f728ce6d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:40.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.582+0000 7f728ce6d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:40.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.582+0000 7f728ce6d640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.582+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f72880c6e30 con 0x7f7288155260 2026-02-22T21:37:40.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.587+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff5ac002ac0 con 0x7ff5b8155e60 2026-02-22T21:37:40.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.588+0000 7ff59effd640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 0x7ff58807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.588+0000 7ff5be548640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 0x7ff58807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.588+0000 7ff5be548640 10 monclient: get_auth_request con 0x7ff588078e00 auth_method 0 2026-02-22T21:37:40.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.588+0000 7ff5be548640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 0x7ff58807b2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.588+0000 7ff59effd640 10 client.4554.objecter ms_handle_connect 0x7ff588078e00 2026-02-22T21:37:40.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7ff5ac05dec0 con 0x7ff5b8155e60 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 10 client.4554.objecter ms_dispatch2 0x7ff5b8000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 3 client.4554.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 3 client.4554.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 20 client.4554.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff5ac062580 con 0x7ff5b8155e60 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.597+0000 7ff59effd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.601+0000 7f65c17fa640 1 -- 10.20.193.51:0/4242465856 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 15}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f65cc0cbf20 con 0x7f65cc1526a0 2026-02-22T21:37:40.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.601+0000 7f65c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:40.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.601+0000 7f65c17fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.602 INFO:teuthology.orchestra.run.trial051.stdout:90194313221 2026-02-22T21:37:40.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 10 librados: watch_flush enter 2026-02-22T21:37:40.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 10 librados: watch_flush exit 2026-02-22T21:37:40.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 20 client.4545.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 10 client.4545.objecter successfully canceled tick 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 msgr2=0x7f659c07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 0x7f659c07b2f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009750 tx=0x7f65bc006ca0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 10 monclient: shutdown 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 msgr2=0x7f65cc14ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f65b4002410 tx=0x7f65b40047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 shutdown_connections 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f659c078e30 0x7f659c07b2f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f65cc152a80 0x7f65cc150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f65cc1526a0 0x7f65cc14ffc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 --2- 10.20.193.51:0/4242465856 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f65cc0cdb20 0x7f65cc141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 >> 10.20.193.51:0/4242465856 conn(0x7f65cc0ce350 msgr2=0x7f65cc0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 shutdown_connections 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 -- 10.20.193.51:0/4242465856 wait complete. 2026-02-22T21:37:40.605 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.603+0000 7f65d2ec9640 1 librados: shutdown 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94002c40 con 0x7fee8c004900 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: dump: 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee94015070 con 0x7fee8c004900 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94004070 con 0x7fee8c004900 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 5 monclient: authenticate success, global_id 4557 2026-02-22T21:37:40.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 10 monclient: _renew_subs 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea815bff0 con 0x7fee8c004900 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea67fc640 4 set_mon_vals no callback set 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feea4ff9640 10 monclient: dump: 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 10 monclient: _renew_subs 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea815fef0 con 0x7fee8c004900 2026-02-22T21:37:40.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.607+0000 7feeae5e4640 1 librados: init done 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.608+0000 7feeae5e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.608+0000 7feeae5e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.608+0000 7feeae5e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.608+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea80cd2b0 con 0x7fee8c004900 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.610+0000 7efd26ffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7efd28152a80 0x7efd280cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47088/0 (socket says 10.20.193.51:47088) 2026-02-22T21:37:40.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.610+0000 7efd26ffd640 1 -- 10.20.193.51:0/340841459 learned_addr learned my addr 10.20.193.51:0/340841459 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:40.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313222 got 90194313221 for osd.15 2026-02-22T21:37:40.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.629+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f8a7000f460 con 0x7f8a880c95a0 2026-02-22T21:37:40.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.629+0000 7f8a677fe640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 0x7f8a5407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.629+0000 7f8a86ffd640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 0x7f8a5407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.629+0000 7f8a86ffd640 10 monclient: get_auth_request con 0x7f8a54078e00 auth_method 0 2026-02-22T21:37:40.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.630+0000 7f8a86ffd640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 0x7f8a5407b2c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009770 tx=0x7f8a7c006cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.630+0000 7f8a677fe640 10 client.4511.objecter ms_handle_connect 0x7f8a54078e00 2026-02-22T21:37:40.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.631+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f8a70097860 con 0x7f8a880c95a0 2026-02-22T21:37:40.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.631+0000 7f8a677fe640 10 client.4511.objecter ms_dispatch2 0x7f8a88000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.631+0000 7f8a677fe640 3 client.4511.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.631+0000 7f8a677fe640 3 client.4511.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.631+0000 7f8a677fe640 20 client.4511.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.638+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f8a7001d020 con 0x7f8a880c95a0 2026-02-22T21:37:40.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.638+0000 7f8a677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.638+0000 7f8a677fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc004480 con 0x7f16d806d9d0 2026-02-22T21:37:40.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7efd277fe640 1 --2- 10.20.193.51:0/340841459 >> v2:10.20.193.125:3300/0 conn(0x7efd281526a0 0x7efd280733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: dump: 2026-02-22T21:37:40.643 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.643 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16cc015070 con 0x7f16d806d9d0 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 5 monclient: authenticate success, global_id 4514 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _renew_subs 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d815a560 con 0x7f16d806d9d0 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc002920 con 0x7f16d806d9d0 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16beffd640 10 monclient: dump: 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.645 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16d4d72640 4 set_mon_vals no callback set 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _renew_subs 2026-02-22T21:37:40.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d815ab20 con 0x7f16d806d9d0 2026-02-22T21:37:40.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 1 librados: init done 2026-02-22T21:37:40.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391000c790 con 0x7f3914155e60 2026-02-22T21:37:40.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: dump: 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3910010070 con 0x7f3914155e60 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391000cb90 con 0x7f3914155e60 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 5 monclient: authenticate success, global_id 4517 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 10 monclient: _renew_subs 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f3918b8c640 4 set_mon_vals no callback set 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391414df20 con 0x7f3914155e60 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f38faffd640 10 monclient: dump: 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 10 monclient: _renew_subs 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39140d06d0 con 0x7f3914155e60 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f391c61a640 1 librados: init done 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7efd277fe640 10 monclient(hunting): get_auth_request con 0x7efd281526a0 auth_method 0 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7efd277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7efd277fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7efd277fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.640+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d8152e90 con 0x7f16d806d9d0 2026-02-22T21:37:40.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f391c61a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f391c61a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f391c61a640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391406e6e0 con 0x7f3914155e60 2026-02-22T21:37:40.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9024062960 con 0x7f90340cdb20 2026-02-22T21:37:40.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f9021ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7f9021ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.643+0000 7efd27fff640 1 --2- 10.20.193.51:0/340841459 >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 0x7efd2815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd27fff640 10 monclient(hunting): get_auth_request con 0x7efd2815d3b0 auth_method 0 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd27fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.645+0000 7efd27fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 1 -- 10.20.193.51:0/340841459 >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 msgr2=0x7efd2815d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 1 --2- 10.20.193.51:0/340841459 >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 0x7efd2815d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 1 -- 10.20.193.51:0/340841459 >> v1:10.20.193.51:6789/0 conn(0x7efd28152a80 legacy=0x7efd280cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd277fe640 1 -- 10.20.193.51:0/340841459 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd28074040 con 0x7efd281526a0 2026-02-22T21:37:40.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.646+0000 7efd27fff640 1 --2- 10.20.193.51:0/340841459 >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 0x7efd2815d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2026-02-22T21:37:40.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.648+0000 7f92ebfff640 1 -- 10.20.193.51:0/2624368064 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 12}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f93100c7300 con 0x7f9310156160 2026-02-22T21:37:40.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.648+0000 7f92ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:40.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.648+0000 7f92ebfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.658 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:40.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 10 librados: watch_flush enter 2026-02-22T21:37:40.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 10 librados: watch_flush exit 2026-02-22T21:37:40.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 20 client.4531.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 10 client.4531.objecter successfully canceled tick 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 msgr2=0x7f92e007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 0x7f92e007b310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92fc004660 tx=0x7f92fc009110 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 10 monclient: shutdown 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 msgr2=0x7f9310150020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f930402f8c0 tx=0x7f9304004b40 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 shutdown_connections 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f92e0078e50 0x7f92e007b310 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9310156160 0x7f9310150020 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9310155d80 0x7f9310158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 --2- 10.20.193.51:0/2624368064 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f93100743d0 0x7f93101f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 >> 10.20.193.51:0/2624368064 conn(0x7f93100ca250 msgr2=0x7f93100ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 shutdown_connections 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 -- 10.20.193.51:0/2624368064 wait complete. 2026-02-22T21:37:40.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.651+0000 7f9315cfc640 1 librados: shutdown 2026-02-22T21:37:40.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.654+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f391001d030 con 0x7f3914155e60 2026-02-22T21:37:40.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.654+0000 7f38faffd640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 0x7f38e407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.655+0000 7f391a38f640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 0x7f38e407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.655+0000 7f391a38f640 10 monclient: get_auth_request con 0x7f38e4078e50 auth_method 0 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.655+0000 7f391a38f640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 0x7f38e407b310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39140cf370 tx=0x7f3904033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.655+0000 7f38faffd640 10 client.4517.objecter ms_handle_connect 0x7f38e4078e50 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.656+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f391000cd30 con 0x7f3914155e60 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.656+0000 7f38faffd640 10 client.4517.objecter ms_dispatch2 0x7f3914000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.656+0000 7f38faffd640 3 client.4517.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.656+0000 7f38faffd640 3 client.4517.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.656+0000 7f38faffd640 20 client.4517.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.658+0000 7efd277fe640 1 --2- 10.20.193.51:0/340841459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd281526a0 0x7efd280733d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd2806cf40 tx=0x7efd1402eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d7b2862cb461e93 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215108 for osd.12 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.678+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337728925 0 0) 0x7f91bc073d60 con 0x7f91bc1526a0 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.678+0000 7f91c1084640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.678+0000 7f91c1084640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.678+0000 7f91c1084640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.678+0000 7f91c1084640 10 monclient(hunting): my global_id is 4561 2026-02-22T21:37:40.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.679+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91a4003620 con 0x7f91bc1526a0 2026-02-22T21:37:40.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.685+0000 7ff5c07d3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:40.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.685+0000 7ff5c07d3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:40.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.685+0000 7ff5c07d3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.685+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 14} v 0) -- 0x7ff5b8153a60 con 0x7ff5b8155e60 2026-02-22T21:37:40.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.723+0000 7f8a8d474640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:40.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.723+0000 7f8a8d474640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:40.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.723+0000 7f8a8d474640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.723+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8a880c6510 con 0x7f8a880c95a0 2026-02-22T21:37:40.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.731+0000 7f903b67c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:40.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.731+0000 7f903b67c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:40.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.731+0000 7f903b67c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.731+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f90340d0540 con 0x7f90340cdb20 2026-02-22T21:37:40.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.733+0000 7f9021ffb640 1 -- 10.20.193.51:0/1101116074 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9024062260 con 0x7f90340cdb20 2026-02-22T21:37:40.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.733+0000 7f9021ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:40.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.733+0000 7f9021ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.735 INFO:teuthology.orchestra.run.trial051.stdout:98784247810 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 10 librados: watch_flush enter 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 10 librados: watch_flush exit 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 20 client.4555.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 10 client.4555.objecter successfully canceled tick 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 msgr2=0x7f900407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 0x7f900407b2c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f903406cf40 tx=0x7f9028031000 comp rx=0 tx=0).stop 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 10 monclient: shutdown 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 msgr2=0x7f90341f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f902400b4d0 tx=0x7f902400b9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 shutdown_connections 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9004078e00 0x7f900407b2c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9034152a80 0x7f903415dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f90341526a0 0x7f90340ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 --2- 10.20.193.51:0/1101116074 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f90340cdb20 0x7f90341f3810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 >> 10.20.193.51:0/1101116074 conn(0x7f90340ce350 msgr2=0x7f90340d1020 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 shutdown_connections 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 -- 10.20.193.51:0/1101116074 wait complete. 2026-02-22T21:37:40.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.735+0000 7f903b67c640 1 librados: shutdown 2026-02-22T21:37:40.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.9 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909333634 0 0) 0x7f91b0002920 con 0x7f91bc152a80 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 10 monclient(hunting): _negotiate 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 10 monclient(hunting): my global_id is 4563 2026-02-22T21:37:40.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.750+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91bc073d60 con 0x7f91bc152a80 2026-02-22T21:37:40.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.757+0000 7ff59effd640 1 -- 10.20.193.51:0/326625139 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 14}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7ff5ac061e80 con 0x7ff5b8155e60 2026-02-22T21:37:40.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.757+0000 7ff59effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:40.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.757+0000 7ff59effd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.760 INFO:teuthology.orchestra.run.trial051.stdout:98784247810 2026-02-22T21:37:40.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 10 librados: watch_flush enter 2026-02-22T21:37:40.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 10 librados: watch_flush exit 2026-02-22T21:37:40.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 20 client.4554.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 10 client.4554.objecter successfully canceled tick 2026-02-22T21:37:40.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 msgr2=0x7ff58807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 0x7ff58807b2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8006fd0 tx=0x7ff5a8006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 10 monclient: shutdown 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 msgr2=0x7ff5b813eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.759+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00b500 tx=0x7ff5ac00b9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 shutdown_connections 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff588078e00 0x7ff58807b2c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff5b815c790 0x7ff5b81f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff5b8155e60 0x7ff5b813eba0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 --2- 10.20.193.51:0/326625139 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff5b80c9470 0x7ff5b81499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 >> 10.20.193.51:0/326625139 conn(0x7ff5b806d8c0 msgr2=0x7ff5b80cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 shutdown_connections 2026-02-22T21:37:40.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 -- 10.20.193.51:0/326625139 wait complete. 2026-02-22T21:37:40.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.760+0000 7ff5c07d3640 1 librados: shutdown 2026-02-22T21:37:40.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.764+0000 7f726effd640 1 -- 10.20.193.51:0/363198135 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7270062050 con 0x7f7288155260 2026-02-22T21:37:40.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.764+0000 7f726effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:40.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.764+0000 7f726effd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f8a677fe640 1 -- 10.20.193.51:0/350490732 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f8a70062680 con 0x7f8a880c95a0 2026-02-22T21:37:40.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f8a677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:40.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f8a677fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.767 INFO:teuthology.orchestra.run.trial051.stdout:103079215107 2026-02-22T21:37:40.767 INFO:teuthology.orchestra.run.trial051.stdout:98784247811 2026-02-22T21:37:40.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f3910061e30 con 0x7f3914155e60 2026-02-22T21:37:40.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f38faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.765+0000 7f38faffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fee94004210 con 0x7fee8c004900 2026-02-22T21:37:40.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 10 librados: watch_flush enter 2026-02-22T21:37:40.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 10 librados: watch_flush enter 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 10 librados: watch_flush exit 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 20 client.4508.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 10 client.4508.objecter successfully canceled tick 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 10 librados: watch_flush exit 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 20 client.4511.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 10 client.4511.objecter successfully canceled tick 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea4ff9640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 0x7fee7807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 msgr2=0x7f725407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 0x7f725407b2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f727c036000 tx=0x7f727c006040 comp rx=0 tx=0).stop 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 10 monclient: shutdown 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 msgr2=0x7f72881ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 msgr2=0x7f8a5407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 0x7f8a5407b2c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009770 tx=0x7f8a7c006cc0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 10 monclient: shutdown 2026-02-22T21:37:40.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 msgr2=0x7f8a881f5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f727000c760 tx=0x7f727000cc30 comp rx=0 tx=0).stop 2026-02-22T21:37:40.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b550 tx=0x7f8a7000ba20 comp rx=0 tx=0).stop 2026-02-22T21:37:40.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 shutdown_connections 2026-02-22T21:37:40.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7254078e00 0x7f725407b2c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f72881556c0 0x7f72881efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7288155260 0x7f72881ef620 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 --2- 10.20.193.51:0/363198135 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7274003ba0 0x7f7288153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 >> 10.20.193.51:0/363198135 conn(0x7f72880cd7b0 msgr2=0x7f72880c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 shutdown_connections 2026-02-22T21:37:40.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8a54078e00 0x7f8a5407b2c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8a88152ab0 0x7f8a881f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8a880c95a0 0x7f8a881f5650 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 --2- 10.20.193.51:0/350490732 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8a8806d9d0 0x7f8a881e8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 >> 10.20.193.51:0/350490732 conn(0x7f8a880cd7b0 msgr2=0x7f8a880cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 shutdown_connections 2026-02-22T21:37:40.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 -- 10.20.193.51:0/363198135 wait complete. 2026-02-22T21:37:40.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 shutdown_connections 2026-02-22T21:37:40.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 -- 10.20.193.51:0/350490732 wait complete. 2026-02-22T21:37:40.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f728ce6d640 1 librados: shutdown 2026-02-22T21:37:40.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea77fe640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 0x7fee7807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea77fe640 10 monclient: get_auth_request con 0x7fee78078e50 auth_method 0 2026-02-22T21:37:40.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7f8a8d474640 1 librados: shutdown 2026-02-22T21:37:40.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea77fe640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 0x7fee7807b310 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee98039000 tx=0x7fee98031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.767+0000 7feea4ff9640 10 client.4557.objecter ms_handle_connect 0x7fee78078e50 2026-02-22T21:37:40.778 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247810 for osd.14 2026-02-22T21:37:40.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215107 for osd.1 2026-02-22T21:37:40.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247811 for osd.10 2026-02-22T21:37:40.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7fee9405dfb0 con 0x7fee8c004900 2026-02-22T21:37:40.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 10 client.4557.objecter ms_dispatch2 0x7feea8000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 3 client.4557.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 3 client.4557.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 20 client.4557.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fee9400d040 con 0x7fee8c004900 2026-02-22T21:37:40.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.776+0000 7feea4ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2860799766 0 0) 0x7f91bc073d60 con 0x7f91bc152a80 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 >> v1:10.20.193.125:6789/0 conn(0x7f91bc15d3b0 legacy=0x7f91bc15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 >> v1:10.20.193.68:6789/0 conn(0x7f91bc1526a0 legacy=0x7f91bc0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc074320 con 0x7f91bc152a80 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f91b00030b0 con 0x7f91bc152a80 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: dump: 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:37:40.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.778+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 >> v1:10.20.193.51:6789/0 conn(0x7f91bc152a80 legacy=0x7f91bc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91a4003e00 0x7f91a4004200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): picked mon.a con 0x7f91a4003e00 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 0x7f91a4004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): picked mon.b con 0x7f91a4004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4009ab0 0x7f91a400bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): picked mon.c con 0x7f91a4009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f91a400c490 con 0x7f91a4003e00 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f91a400c610 con 0x7f91a4004900 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f91a400c810 con 0x7f91a4009ab0 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1084640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1885640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 0x7f91a4004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1885640 10 monclient(hunting): get_auth_request con 0x7f91a4004900 auth_method 0 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1885640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.779+0000 7f91c1885640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.780+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.780+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.780+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.780+0000 7f91c2086640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91a4003e00 0x7f91a4004200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c2086640 10 monclient(hunting): get_auth_request con 0x7f91a4003e00 auth_method 0 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c2086640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c2086640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c2086640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 10 monclient(hunting): handle_auth_done global_id 4563 payload 293 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 -- 10.20.193.51:0/1312805412 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4009ab0 msgr2=0x7f91a400bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4009ab0 0x7f91a400bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 -- 10.20.193.51:0/1312805412 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91a4003e00 msgr2=0x7f91a4004200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91a4003e00 0x7f91a4004200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 monclient: found mon.b 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 1 -- 10.20.193.51:0/1312805412 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc073d60 con 0x7f91a4004900 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c1885640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c4311640 5 monclient: authenticate success, global_id 4563 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c4311640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.781+0000 7f91c2887640 1 -- 10.20.193.51:0/1312805412 reap_dead start 2026-02-22T21:37:40.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.782+0000 7f91c1885640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 0x7f91a4004d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f91b000c270 tx=0x7f91b002f590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7eff4c6f5a266fb server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0007d80 con 0x7f91a4004900 2026-02-22T21:37:40.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: dump: 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c4311640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b0006af0 con 0x7f91a4004900 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 1 -- 10.20.193.51:0/1312805412 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0007030 con 0x7f91a4004900 2026-02-22T21:37:40.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c1084640 10 monclient: dump: 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c4311640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.783+0000 7f91c4311640 4 set_mon_vals no callback set 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 10 monclient: shutdown 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 -- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 msgr2=0x7f91a4004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 0x7f91a4004d80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f91b000c270 tx=0x7f91b002f590 comp rx=0 tx=0).stop 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 -- 10.20.193.51:0/1312805412 shutdown_connections 2026-02-22T21:37:40.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 --2- 10.20.193.51:0/1312805412 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91a4004900 0x7f91a4004d80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 -- 10.20.193.51:0/1312805412 >> 10.20.193.51:0/1312805412 conn(0x7f91bc0ce350 msgr2=0x7f91bc0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 -- 10.20.193.51:0/1312805412 shutdown_connections 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 1 -- 10.20.193.51:0/1312805412 wait complete. 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.784+0000 7f91c4311640 10 monclient: monmap: 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 librados: starting msgr at 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 librados: starting objecter 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 Processor -- start 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 -- start start 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 librados: setting wanted keys 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 librados: calling monclient init 2026-02-22T21:37:40.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient: init 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4004900 0x7f91bc1f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-c con 0x7f91a4004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 0x7f91bc1f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-b con 0x7f91bc0d0d10 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 0x7f91bc14a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): picked mon.noname-a con 0x7f91bc1f7b30 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc074c50 con 0x7f91bc1f7b30 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc0cca00 con 0x7f91bc0d0d10 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc0748e0 con 0x7f91a4004900 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c4311640 10 monclient(hunting): authenticate will time out at 1136.365687s 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.785+0000 7f91c2887640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 0x7f91bc14a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 0x7f91bc1f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 0x7f91bc14a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33178/0 (socket says 10.20.193.51:33178) 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 1 -- 10.20.193.51:0/100561024 learned_addr learned my addr 10.20.193.51:0/100561024 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): get_auth_request con 0x7f91bc1f7b30 auth_method 0 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): get_auth_request con 0x7f91bc0d0d10 auth_method 0 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c2887640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.786+0000 7f91c1885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 10 monclient(hunting): handle_auth_done global_id 4564 payload 824 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 -- 10.20.193.51:0/100561024 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4004900 msgr2=0x7f91bc1f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4004900 0x7f91bc1f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 -- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 msgr2=0x7f91bc14a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 0x7f91bc14a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 monclient: found mon.noname-b 2026-02-22T21:37:40.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.787+0000 7f91c1885640 1 -- 10.20.193.51:0/100561024 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc0cf6b0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.789+0000 7f91c1885640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 0x7f91bc1f75f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c7e0 tx=0x7f91bc0bbb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.789+0000 7f91a2ffd640 10 client.?.objecter ms_handle_connect 0x7f91bc0d0d10 2026-02-22T21:37:40.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.789+0000 7f91a2ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b0048070 con 0x7f91bc0d0d10 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: dump: 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b003d040 con 0x7f91bc0d0d10 2026-02-22T21:37:40.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b00054c0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91a2ffd640 10 monclient: dump: 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c0883640 4 set_mon_vals no callback set 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 5 monclient: authenticate success, global_id 4564 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 10 monclient: _renew_subs 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91bc0cf990 con 0x7f91bc0d0d10 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.790+0000 7f91c4311640 10 monclient: _renew_subs 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91bc0cff50 con 0x7f91bc0d0d10 2026-02-22T21:37:40.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 1 librados: init done 2026-02-22T21:37:40.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.791+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91bc0cb9d0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f91b00036d0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91a2ffd640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 0x7f918c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91c2086640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 0x7f918c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91c2086640 10 monclient: get_auth_request con 0x7f918c078e50 auth_method 0 2026-02-22T21:37:40.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91a2ffd640 10 client.4564.objecter ms_handle_connect 0x7f918c078e50 2026-02-22T21:37:40.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.802+0000 7f91c2086640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 0x7f918c07b310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f91ac007fd0 tx=0x7f91ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.804+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f91b0081fd0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.804+0000 7f91a2ffd640 10 client.4564.objecter ms_dispatch2 0x7f91bc000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.804+0000 7f91a2ffd640 3 client.4564.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.804+0000 7f91a2ffd640 3 client.4564.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.804+0000 7f91a2ffd640 20 client.4564.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.851+0000 7f391c61a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:40.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.851+0000 7f391c61a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:40.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.851+0000 7f391c61a640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.851+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f39140cede0 con 0x7f3914155e60 2026-02-22T21:37:40.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.861+0000 7feeae5e4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:40.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.861+0000 7feeae5e4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:40.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.861+0000 7feeae5e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:40.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.861+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 13} v 0) -- 0x7feea814f180 con 0x7fee8c004900 2026-02-22T21:37:40.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.863+0000 7feea4ff9640 1 -- 10.20.193.51:0/2149629945 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 13}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7fee94062670 con 0x7fee8c004900 2026-02-22T21:37:40.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.863+0000 7feea4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:40.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.863+0000 7feea4ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.865 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 10 librados: watch_flush enter 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 10 librados: watch_flush exit 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 20 client.4557.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 10 client.4557.objecter successfully canceled tick 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 msgr2=0x7fee7807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 0x7fee7807b310 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee98039000 tx=0x7fee98031000 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 10 monclient: shutdown 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 msgr2=0x7feea815bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.864+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fee9400b450 tx=0x7fee9400b920 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 shutdown_connections 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fee78078e50 0x7fee7807b310 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feea815d640 0x7feea815a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feea8143b20 0x7feea815d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 --2- 10.20.193.51:0/2149629945 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fee8c004900 0x7feea815bab0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 >> 10.20.193.51:0/2149629945 conn(0x7feea806d8c0 msgr2=0x7feea8153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 shutdown_connections 2026-02-22T21:37:40.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 -- 10.20.193.51:0/2149629945 wait complete. 2026-02-22T21:37:40.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.865+0000 7feeae5e4640 1 librados: shutdown 2026-02-22T21:37:40.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247812 for osd.13 2026-02-22T21:37:40.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 1 -- 10.20.193.51:0/340841459 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1403c070 con 0x7efd281526a0 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: dump: 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.901 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 1 -- 10.20.193.51:0/340841459 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1402fb80 con 0x7efd281526a0 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 1 -- 10.20.193.51:0/340841459 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14037440 con 0x7efd281526a0 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd2e032640 5 monclient: authenticate success, global_id 4520 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd2e032640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd2e032640 4 set_mon_vals no callback set 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.898+0000 7efd267fc640 10 monclient: dump: 2026-02-22T21:37:40.902 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 10 monclient: shutdown 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 -- 10.20.193.51:0/340841459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd281526a0 msgr2=0x7efd280733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 --2- 10.20.193.51:0/340841459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd281526a0 0x7efd280733d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efd2806cf40 tx=0x7efd1402eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 -- 10.20.193.51:0/340841459 shutdown_connections 2026-02-22T21:37:40.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 --2- 10.20.193.51:0/340841459 >> v2:10.20.193.68:3300/0 conn(0x7efd2815d3b0 0x7efd2815d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 --2- 10.20.193.51:0/340841459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd281526a0 0x7efd280733d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 -- 10.20.193.51:0/340841459 >> 10.20.193.51:0/340841459 conn(0x7efd280ce350 msgr2=0x7efd280ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 -- 10.20.193.51:0/340841459 shutdown_connections 2026-02-22T21:37:40.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 1 -- 10.20.193.51:0/340841459 wait complete. 2026-02-22T21:37:40.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 10 monclient: build_initial_monmap 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.899+0000 7efd2e032640 10 monclient: monmap: 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 librados: starting msgr at 2026-02-22T21:37:40.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 librados: starting objecter 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 Processor -- start 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 -- start start 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 librados: setting wanted keys 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 librados: calling monclient init 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient: init 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 0x7efd281591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-a con 0x7efd281526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd28152a80 0x7efd281f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-b con 0x7efd28152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 0x7efd281567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): picked mon.noname-c con 0x7efd2815d3b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7efd280748e0 con 0x7efd281526a0 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7efd28074600 con 0x7efd28152a80 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7efd280cdcf0 con 0x7efd2815d3b0 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.900+0000 7efd2e032640 10 monclient(hunting): authenticate will time out at 1136.480741s 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.901+0000 7efd277fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 0x7efd281591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 0x7efd281591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33194/0 (socket says 10.20.193.51:33194) 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 1 -- 10.20.193.51:0/4104889826 learned_addr learned my addr 10.20.193.51:0/4104889826 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 10 monclient(hunting): get_auth_request con 0x7efd281526a0 auth_method 0 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.902+0000 7efd277fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.903+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.903+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.903+0000 7efd277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f16cc010480 con 0x7f16d806d9d0 2026-02-22T21:37:40.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f16beffd640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 0x7f16a407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f16d5d74640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 0x7f16a407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f16d5d74640 10 monclient: get_auth_request con 0x7f16a4078e50 auth_method 0 2026-02-22T21:37:40.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f91bc0cb9d0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f91a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.906+0000 7f91a2ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 10 client.4514.objecter ms_handle_connect 0x7f16a4078e50 2026-02-22T21:37:40.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16d5d74640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 0x7f16a407b310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f16d80cae50 tx=0x7f16c0031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7f16cc05e100 con 0x7f16d806d9d0 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 10 client.4514.objecter ms_dispatch2 0x7f16d8000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 3 client.4514.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 3 client.4514.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.907+0000 7f16beffd640 20 client.4514.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.910+0000 7efd26ffd640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd28152a80 0x7efd281f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.914+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f16cc0627c0 con 0x7f16d806d9d0 2026-02-22T21:37:40.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.914+0000 7f16beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.914+0000 7f16beffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:40.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 0x7efd281567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7f38faffd640 1 -- 10.20.193.51:0/1326221703 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f3910061730 con 0x7f3914155e60 2026-02-22T21:37:40.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7f38faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:40.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7f38faffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): get_auth_request con 0x7efd2815d3b0 auth_method 0 2026-02-22T21:37:40.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:40.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:40.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:40.925 INFO:teuthology.orchestra.run.trial051.stdout:94489280516 2026-02-22T21:37:40.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.920+0000 7efd27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.921+0000 7efd27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 -- 10.20.193.51:0/4104889826 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd28152a80 msgr2=0x7efd281f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd28152a80 0x7efd281f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 -- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 msgr2=0x7efd281591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 0x7efd281591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 monclient: found mon.noname-c 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7efd27fff640 1 -- 10.20.193.51:0/4104889826 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd28144270 con 0x7efd2815d3b0 2026-02-22T21:37:40.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 10 librados: watch_flush enter 2026-02-22T21:37:40.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 10 librados: watch_flush exit 2026-02-22T21:37:40.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 20 client.4517.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 10 client.4517.objecter successfully canceled tick 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 msgr2=0x7f38e407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 0x7f38e407b310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39140cf370 tx=0x7f3904033000 comp rx=0 tx=0).stop 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 10 monclient: shutdown 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 msgr2=0x7f39140d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f391000b970 tx=0x7f391000be40 comp rx=0 tx=0).stop 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 shutdown_connections 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f38e4078e50 0x7f38e407b310 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3914155e60 0x7f39140d1490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f39140cb570 0x7f391415eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 --2- 10.20.193.51:0/1326221703 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39140c9470 0x7f39141f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 >> 10.20.193.51:0/1326221703 conn(0x7f391406d8c0 msgr2=0x7f3914153680 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 shutdown_connections 2026-02-22T21:37:40.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 -- 10.20.193.51:0/1326221703 wait complete. 2026-02-22T21:37:40.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.922+0000 7f391c61a640 1 librados: shutdown 2026-02-22T21:37:40.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.924+0000 7efd27fff640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 0x7efd281567b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd1c007e90 tx=0x7efd1c007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.924+0000 7efd03fff640 10 client.?.objecter ms_handle_connect 0x7efd2815d3b0 2026-02-22T21:37:40.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.924+0000 7efd03fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:40.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c01a070 con 0x7efd2815d3b0 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: dump: 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.935 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c003a90 con 0x7efd2815d3b0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c003e60 con 0x7efd2815d3b0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 5 monclient: authenticate success, global_id 4526 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _renew_subs 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd25ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd281f7550 con 0x7efd2815d3b0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd03fff640 10 monclient: dump: 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:40.936 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _renew_subs 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd281f7b10 con 0x7efd2815d3b0 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 1 librados: init done 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:40.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:40.938 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.926+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2815d820 con 0x7efd2815d3b0 2026-02-22T21:37:40.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.11 2026-02-22T21:37:40.939 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.937+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7efd1c004030 con 0x7efd2815d3b0 2026-02-22T21:37:40.939 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.937+0000 7efd03fff640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 0x7efcf807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:40.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.938+0000 7efd277fe640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 0x7efcf807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:40.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.938+0000 7efd277fe640 10 monclient: get_auth_request con 0x7efcf8078e30 auth_method 0 2026-02-22T21:37:40.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.938+0000 7efd277fe640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 0x7efcf807b2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efd1402eef0 tx=0x7efd14033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:40.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.938+0000 7efd03fff640 10 client.4526.objecter ms_handle_connect 0x7efcf8078e30 2026-02-22T21:37:40.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.939+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 8972+0+0 (secure 0 0 0) 0x7efd1c05e380 con 0x7efd2815d3b0 2026-02-22T21:37:40.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.939+0000 7efd03fff640 10 client.4526.objecter ms_dispatch2 0x7efd28000b90 osd_map(30..30 src has 1..30) 2026-02-22T21:37:40.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.939+0000 7efd03fff640 3 client.4526.objecter handle_osd_map got epochs [30,30] > 0 2026-02-22T21:37:40.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.939+0000 7efd03fff640 3 client.4526.objecter handle_osd_map decoding full epoch 30 2026-02-22T21:37:40.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.939+0000 7efd03fff640 20 client.4526.objecter dump_active .. 0 homeless 2026-02-22T21:37:40.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.992+0000 7f91c4311640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:40.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.992+0000 7f91c4311640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:40.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.992+0000 7f91c4311640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:40.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.992+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f91bc15d820 con 0x7f91bc0d0d10 2026-02-22T21:37:40.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.994+0000 7f91a2ffd640 1 -- 10.20.193.51:0/100561024 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f91bc073ae0 con 0x7f91bc0d0d10 2026-02-22T21:37:40.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.994+0000 7f91a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2026-02-22T21:37:40.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.994+0000 7f91a2ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:40.996 INFO:teuthology.orchestra.run.trial051.stdout:94489280517 2026-02-22T21:37:40.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 10 librados: watch_flush enter 2026-02-22T21:37:40.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 10 librados: watch_flush exit 2026-02-22T21:37:40.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 20 client.4564.objecter shutdown clearing up homeless session... 2026-02-22T21:37:40.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 10 client.4564.objecter successfully canceled tick 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 msgr2=0x7f918c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 0x7f918c07b310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f91ac007fd0 tx=0x7f91ac009040 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 10 monclient: shutdown 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 msgr2=0x7f91bc1f75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 0x7f91bc1f75f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c7e0 tx=0x7f91bc0bbb80 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 shutdown_connections 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f918c078e50 0x7f918c07b310 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f91bc1f7b30 0x7f91bc14a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f91bc0d0d10 0x7f91bc1f75f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 --2- 10.20.193.51:0/100561024 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f91a4004900 0x7f91bc1f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 >> 10.20.193.51:0/100561024 conn(0x7f91bc0ce350 msgr2=0x7f91bc14ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 shutdown_connections 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 -- 10.20.193.51:0/100561024 wait complete. 2026-02-22T21:37:40.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:40.996+0000 7f91c4311640 1 librados: shutdown 2026-02-22T21:37:41.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.000+0000 7f16dceda640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:41.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.000+0000 7f16dceda640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:41.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.000+0000 7f16dceda640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.000+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f16d8152090 con 0x7f16d806d9d0 2026-02-22T21:37:41.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.7 2026-02-22T21:37:41.007 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.040+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7efd1c024020 con 0x7efd2815d3b0 2026-02-22T21:37:41.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.040+0000 7efd03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.040+0000 7efd03fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.073+0000 7f16beffd640 1 -- 10.20.193.51:0/1322161604 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f16cc0620c0 con 0x7f16d806d9d0 2026-02-22T21:37:41.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.073+0000 7f16beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2026-02-22T21:37:41.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.073+0000 7f16beffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.075 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 10 librados: watch_flush enter 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 10 librados: watch_flush exit 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 20 client.4514.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 10 client.4514.objecter successfully canceled tick 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 msgr2=0x7f16a407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 0x7f16a407b310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f16d80cae50 tx=0x7f16c0031000 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 10 monclient: shutdown 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 msgr2=0x7f16d8159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b4d0 tx=0x7f16cc00b9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 shutdown_connections 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16a4078e50 0x7f16a407b310 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16d8152ab0 0x7f16d81e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16d80c93d0 0x7f16d81487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 --2- 10.20.193.51:0/1322161604 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16d806d9d0 0x7f16d8159d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 >> 10.20.193.51:0/1322161604 conn(0x7f16d80cd7b0 msgr2=0x7f16d80cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 shutdown_connections 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 -- 10.20.193.51:0/1322161604 wait complete. 2026-02-22T21:37:41.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.075+0000 7f16dceda640 1 librados: shutdown 2026-02-22T21:37:41.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.4 2026-02-22T21:37:41.085 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.123+0000 7efd2e032640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:41.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.123+0000 7efd2e032640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:41.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.123+0000 7efd2e032640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.123+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7efd280cb0f0 con 0x7efd2815d3b0 2026-02-22T21:37:41.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.125+0000 7efd03fff640 1 -- 10.20.193.51:0/4104889826 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efd1c010070 con 0x7efd2815d3b0 2026-02-22T21:37:41.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.125+0000 7efd03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2026-02-22T21:37:41.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.125+0000 7efd03fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.126 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 10 librados: watch_flush enter 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 10 librados: watch_flush exit 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 20 client.4526.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 10 client.4526.objecter successfully canceled tick 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 msgr2=0x7efcf807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 0x7efcf807b2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efd1402eef0 tx=0x7efd14033000 comp rx=0 tx=0).stop 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 10 monclient: shutdown 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 msgr2=0x7efd281567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 0x7efd281567b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd1c007e90 tx=0x7efd1c007f90 comp rx=0 tx=0).stop 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 shutdown_connections 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efcf8078e30 0x7efcf807b2f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efd2815d3b0 0x7efd281567b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efd28152a80 0x7efd281f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 --2- 10.20.193.51:0/4104889826 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efd281526a0 0x7efd281591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 >> 10.20.193.51:0/4104889826 conn(0x7efd280ce350 msgr2=0x7efd281518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 shutdown_connections 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 -- 10.20.193.51:0/4104889826 wait complete. 2026-02-22T21:37:41.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.126+0000 7efd2e032640 1 librados: shutdown 2026-02-22T21:37:41.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.5 2026-02-22T21:37:41.136 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.287 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:12.441 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.288 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:12.442 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.312 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:12.465 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.314 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:12.430 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.314 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:12.466 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.315 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:12.467 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.315 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:12.431 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.339 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:12.455 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.339 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:12.491 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:37:41.392 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2026-02-22T21:37:41.401 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2026-02-22T21:37:41.455 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T21:37:41.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.462 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 10 monclient: monmap: 2026-02-22T21:37:41.462 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.462 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.460734+0000 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.460734+0000 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.463 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 1 Processor -- start 2026-02-22T21:37:41.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 1 -- start start 2026-02-22T21:37:41.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.460+0000 7f93773e3640 10 monclient: init 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient: _add_conns ranks=[2,4,3,1,5,0] 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f937015f710 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f937015faf0 0x7f937015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-b con 0x7f937015faf0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f937015d8e0 0x7f93700ca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-a con 0x7f937015d8e0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937015dd50 con 0x7f937015f710 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f93700cbb90 con 0x7f937015d8e0 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f93700caca0 con 0x7f937015faf0 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f93773e3640 10 monclient(hunting): authenticate will time out at 1137.041235s 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375959640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f937015d8e0 0x7f93700ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9374957640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f937015faf0 0x7f937015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375158640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f937015f710 0x7f93700700e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33276/0 (socket says 10.20.193.51:33276) 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375158640 1 -- 10.20.193.51:0/4021229837 learned_addr learned my addr 10.20.193.51:0/4021229837 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375959640 10 monclient(hunting): get_auth_request con 0x7f937015d8e0 auth_method 0 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375959640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375959640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.461+0000 7f9375959640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): get_auth_request con 0x7f937015faf0 auth_method 0 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.462+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 10 monclient(hunting): handle_auth_done global_id 4567 payload 274 2026-02-22T21:37:41.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f935ffff640 1 -- 10.20.193.51:0/4021229837 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2821915884 0 0) 0x7f937015dd50 con 0x7f937015f710 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 1 -- 10.20.193.51:0/4021229837 >> v2:10.20.193.51:3300/0 conn(0x7f937015d8e0 msgr2=0x7f93700ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 1 --2- 10.20.193.51:0/4021229837 >> v2:10.20.193.51:3300/0 conn(0x7f937015d8e0 0x7f93700ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 1 -- 10.20.193.51:0/4021229837 >> v1:10.20.193.125:6789/0 conn(0x7f937015f710 legacy=0x7f93700700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f9374957640 1 -- 10.20.193.51:0/4021229837 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93700caf80 con 0x7f937015faf0 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.463+0000 7f935ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2026-02-22T21:37:41.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.464+0000 7f9374957640 1 --2- 10.20.193.51:0/4021229837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f937015d3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9370072e40 tx=0x7f936402ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8d5984d3ac47a5 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 1 -- 10.20.193.51:0/4021229837 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936403c070 con 0x7f937015faf0 2026-02-22T21:37:41.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: dump: 2026-02-22T21:37:41.482 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.482 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.482 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.483 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 1 -- 10.20.193.51:0/4021229837 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936402faf0 con 0x7f937015faf0 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 1 -- 10.20.193.51:0/4021229837 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364037440 con 0x7f937015faf0 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 5 monclient: authenticate success, global_id 4567 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 4 set_mon_vals no callback set 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f935ffff640 10 monclient: dump: 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.486 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 10 monclient: shutdown 2026-02-22T21:37:41.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 -- 10.20.193.51:0/4021229837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 msgr2=0x7f937015d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 --2- 10.20.193.51:0/4021229837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f937015d3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9370072e40 tx=0x7f936402ede0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 -- 10.20.193.51:0/4021229837 shutdown_connections 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 --2- 10.20.193.51:0/4021229837 >> v2:10.20.193.51:3300/0 conn(0x7f937015d8e0 0x7f93700ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 --2- 10.20.193.51:0/4021229837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f937015d3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 -- 10.20.193.51:0/4021229837 >> 10.20.193.51:0/4021229837 conn(0x7f93700cf6c0 msgr2=0x7f93700cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 -- 10.20.193.51:0/4021229837 shutdown_connections 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.466+0000 7f93773e3640 1 -- 10.20.193.51:0/4021229837 wait complete. 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient: monmap: 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.489 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.490 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 librados: starting msgr at 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 librados: starting objecter 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 Processor -- start 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 -- start start 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 librados: setting wanted keys 2026-02-22T21:37:41.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 librados: calling monclient init 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient: init 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f937015d8e0 0x7f937015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-c con 0x7f937015d8e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-a con 0x7f937015f710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f9370151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): picked mon.noname-b con 0x7f937015faf0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f93700cb820 con 0x7f937015f710 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f93700cb540 con 0x7f937015faf0 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f93700707f0 con 0x7f937015d8e0 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.467+0000 7f93773e3640 10 monclient(hunting): authenticate will time out at 1137.047820s 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f937015d8e0 0x7f937015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33226/0 (socket says 10.20.193.51:33226) 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 1 -- 10.20.193.51:0/2781007233 learned_addr learned my addr 10.20.193.51:0/2781007233 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): get_auth_request con 0x7f937015f710 auth_method 0 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f9370151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): get_auth_request con 0x7f937015d8e0 auth_method 0 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9374957640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): get_auth_request con 0x7f937015faf0 auth_method 0 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375158640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.468+0000 7f9375959640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 10 monclient(hunting): handle_auth_done global_id 4575 payload 824 2026-02-22T21:37:41.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 -- 10.20.193.51:0/2781007233 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f937015d8e0 msgr2=0x7f937015c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f937015d8e0 0x7f937015c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 -- 10.20.193.51:0/2781007233 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 msgr2=0x7f9370151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f9370151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.469+0000 7f9374957640 1 -- 10.20.193.51:0/2781007233 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937015c9a0 con 0x7f937015f710 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.471+0000 7f9374957640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9364009130 tx=0x7f93640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.471+0000 7f935dffb640 10 client.?.objecter ms_handle_connect 0x7f937015f710 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.471+0000 7f935dffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936403c040 con 0x7f937015f710 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: dump: 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.499 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 5 monclient: authenticate success, global_id 4575 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9364042610 con 0x7f937015f710 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _renew_subs 2026-02-22T21:37:41.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f937015b0e0 con 0x7f937015f710 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93640429e0 con 0x7f937015f710 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935f7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f935dffb640 10 monclient: dump: 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _renew_subs 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f937015b6a0 con 0x7f937015f710 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 1 librados: init done 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.473+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f937015d3a0 con 0x7f937015f710 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f936404b450 con 0x7f937015f710 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f935dffb640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 0x7f934007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f9375158640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 0x7f934007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f9375158640 10 monclient: get_auth_request con 0x7f9340078e50 auth_method 0 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f9375158640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 0x7f934007b310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9360007b40 tx=0x7f9360007a90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.488+0000 7f935dffb640 10 client.4575.objecter ms_handle_connect 0x7f9340078e50 2026-02-22T21:37:41.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 10 monclient: monmap: 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.502712+0000 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.502712+0000 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 1 Processor -- start 2026-02-22T21:37:41.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 1 -- start start 2026-02-22T21:37:41.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.502+0000 7fc399cf5640 10 monclient: init 2026-02-22T21:37:41.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient: _add_conns ranks=[4,2,0,5,3,1] 2026-02-22T21:37:41.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc3941526a0 0x7fc3940733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-b con 0x7fc3941526a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc394152a80 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc39415d3b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc394073d60 con 0x7fc39415d3b0 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc394074040 con 0x7fc394152a80 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc3940cca00 con 0x7fc3941526a0 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc399cf5640 10 monclient(hunting): authenticate will time out at 1137.083109s 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc392ffd640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fc394152a80 0x7fc3940cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33286/0 (socket says 10.20.193.51:33286) 2026-02-22T21:37:41.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.503+0000 7fc392ffd640 1 -- 10.20.193.51:0/1186299429 learned_addr learned my addr 10.20.193.51:0/1186299429 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768282623 0 0) 0x7fc394074040 con 0x7fc394152a80 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 10 monclient(hunting): my global_id is 4535 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.504+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc378003620 con 0x7fc394152a80 2026-02-22T21:37:41.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.505+0000 7fc3937fe640 1 --2- 10.20.193.51:0/1186299429 >> v2:10.20.193.68:3300/0 conn(0x7fc3941526a0 0x7fc3940733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2303273074 0 0) 0x7fc378003620 con 0x7fc394152a80 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 >> v2:10.20.193.68:3300/0 conn(0x7fc3941526a0 msgr2=0x7fc3940733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 --2- 10.20.193.51:0/1186299429 >> v2:10.20.193.68:3300/0 conn(0x7fc3941526a0 0x7fc3940733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 >> v1:10.20.193.51:6789/0 conn(0x7fc39415d3b0 legacy=0x7fc39415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:41.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc394074320 con 0x7fc394152a80 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fc384002f00 con 0x7fc394152a80 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: dump: 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.509 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 >> v1:10.20.193.125:6789/0 conn(0x7fc394152a80 legacy=0x7fc3940cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc378003ba0 0x7fc378003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): picked mon.b con 0x7fc378003ba0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc378004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): picked mon.c con 0x7fc378004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc378005470 0x7fc37800bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): picked mon.a con 0x7fc378005470 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc3940cca00 con 0x7fc378005470 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc37800c3c0 con 0x7fc378003ba0 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc37800c5c0 con 0x7fc378004900 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.506+0000 7fc3927fc640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:41.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.508+0000 7fc392ffd640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc378004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.508+0000 7fc392ffd640 10 monclient(hunting): get_auth_request con 0x7fc378004900 auth_method 0 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.508+0000 7fc392ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.508+0000 7fc392ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.508+0000 7fc392ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc378003ba0 0x7fc378003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc392ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc392ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc392ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): get_auth_request con 0x7fc378003ba0 auth_method 0 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.509+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 10 monclient(hunting): handle_auth_done global_id 4535 payload 293 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 -- 10.20.193.51:0/1186299429 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc378003ba0 msgr2=0x7fc378003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.512 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc378003ba0 0x7fc378003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 -- 10.20.193.51:0/1186299429 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc378005470 msgr2=0x7fc37800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc378005470 0x7fc37800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 monclient: found mon.c 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 1 -- 10.20.193.51:0/1186299429 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc378003620 con 0x7fc378004900 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc393fff640 1 -- 10.20.193.51:0/1186299429 reap_dead start 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc399cf5640 5 monclient: authenticate success, global_id 4535 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.510+0000 7fc399cf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.513 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.511+0000 7fc392ffd640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc378004d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c2a0 tx=0x7fc38402f5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc53f10be4a0a66e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.520 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T21:37:41.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc384007de0 con 0x7fc378004900 2026-02-22T21:37:41.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: dump: 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc384005bd0 con 0x7fc378004900 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 1 -- 10.20.193.51:0/1186299429 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc384007070 con 0x7fc378004900 2026-02-22T21:37:41.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 4 set_mon_vals no callback set 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc3927fc640 10 monclient: dump: 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 10 monclient: shutdown 2026-02-22T21:37:41.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 -- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 msgr2=0x7fc378004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc378004d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c2a0 tx=0x7fc38402f5d0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 -- 10.20.193.51:0/1186299429 shutdown_connections 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 --2- 10.20.193.51:0/1186299429 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc378004d80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 -- 10.20.193.51:0/1186299429 >> 10.20.193.51:0/1186299429 conn(0x7fc3940ce350 msgr2=0x7fc3940ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 -- 10.20.193.51:0/1186299429 shutdown_connections 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.520+0000 7fc399cf5640 1 -- 10.20.193.51:0/1186299429 wait complete. 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient: monmap: 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.526 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 librados: starting msgr at 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 librados: starting objecter 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 Processor -- start 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 -- start start 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 librados: setting wanted keys 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 librados: calling monclient init 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient: init 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc394159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-c con 0x7fc378004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3941592c0 0x7fc3941f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-a con 0x7fc3941592c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): picked mon.noname-b con 0x7fc39415c4c0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc3940748e0 con 0x7fc3941592c0 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc394074c50 con 0x7fc39415c4c0 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc39415d820 con 0x7fc378004900 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.521+0000 7fc399cf5640 10 monclient(hunting): authenticate will time out at 1137.101697s 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35994/0 (socket says 10.20.193.51:35994) 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 1 -- 10.20.193.51:0/2152880240 learned_addr learned my addr 10.20.193.51:0/2152880240 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): get_auth_request con 0x7fc39415c4c0 auth_method 0 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc3937fe640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc394159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.522+0000 7fc393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): get_auth_request con 0x7fc378004900 auth_method 0 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.523+0000 7fc3937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 10 monclient(hunting): handle_auth_done global_id 4573 payload 824 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 -- 10.20.193.51:0/2152880240 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 msgr2=0x7fc394159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc394159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 -- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3941592c0 msgr2=0x7fc3941f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3941592c0 0x7fc3941f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.524+0000 7fc393fff640 1 -- 10.20.193.51:0/2152880240 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3941f4a10 con 0x7fc39415c4c0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.525+0000 7fc36ffff640 10 client.?.objecter ms_handle_connect 0x7fc39415c4c0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.525+0000 7fc393fff640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3940cb660 tx=0x7fc38800cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.525+0000 7fc36ffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38800aa90 con 0x7fc39415c4c0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: dump: 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.532 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38800ac50 con 0x7fc39415c4c0 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 5 monclient: authenticate success, global_id 4573 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388014420 con 0x7fc39415c4c0 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc391ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc36ffff640 10 monclient: dump: 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.533 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _renew_subs 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39415a390 con 0x7fc39415c4c0 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _renew_subs 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39415a950 con 0x7fc39415c4c0 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 1 librados: init done 2026-02-22T21:37:41.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.527+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39415dbb0 con 0x7fc39415c4c0 2026-02-22T21:37:41.535 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.528+0000 7f70c7a57640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.528+0000 7f70c7a57640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.528+0000 7f70c7a57640 10 monclient: monmap: 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.529401+0000 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.529401+0000 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.538 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 Processor -- start 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 -- start start 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient: init 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient: _add_conns ranks=[5,2,4,0,3,1] 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f70c01526a0 0x7f70c00733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-c con 0x7f70c01526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70c0152a80 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f70c015d3b0 0x7f70c015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-b con 0x7f70c015d3b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c0073d60 con 0x7f70c0152a80 2026-02-22T21:37:41.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f70c00cca00 con 0x7f70c015d3b0 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f70c0074c50 con 0x7f70c01526a0 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.529+0000 7f70c7a57640 10 monclient(hunting): authenticate will time out at 1137.109933s 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c57cc640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f70c01526a0 0x7f70c00733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c4fcb640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f70c0152a80 0x7f70c00cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33300/0 (socket says 10.20.193.51:33300) 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c4fcb640 1 -- 10.20.193.51:0/2404594734 learned_addr learned my addr 10.20.193.51:0/2404594734 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c57cc640 10 monclient(hunting): get_auth_request con 0x7f70c01526a0 auth_method 0 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c57cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c57cc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.530+0000 7f70c57cc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.531+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.531+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.531+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 10 monclient(hunting): handle_auth_done global_id 4541 payload 274 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 1 -- 10.20.193.51:0/2404594734 >> v2:10.20.193.68:3300/0 conn(0x7f70c015d3b0 msgr2=0x7f70c015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 1 --2- 10.20.193.51:0/2404594734 >> v2:10.20.193.68:3300/0 conn(0x7f70c015d3b0 0x7f70c015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 1 -- 10.20.193.51:0/2404594734 >> v1:10.20.193.125:6789/0 conn(0x7f70c0152a80 legacy=0x7f70c00cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 1 monclient: found mon.noname-c 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.532+0000 7f70c57cc640 1 -- 10.20.193.51:0/2404594734 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c0074040 con 0x7f70c01526a0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.533+0000 7f70c57cc640 1 --2- 10.20.193.51:0/2404594734 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c00733d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f70c006cf40 tx=0x7f70b002eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec883a1e155992 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 1 -- 10.20.193.51:0/2404594734 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b003c070 con 0x7f70c01526a0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: dump: 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 1 -- 10.20.193.51:0/2404594734 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b002fb80 con 0x7f70c01526a0 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 1 -- 10.20.193.51:0/2404594734 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b0037440 con 0x7f70c01526a0 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 5 monclient: authenticate success, global_id 4541 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 4 set_mon_vals no callback set 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70affff640 10 monclient: dump: 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.542 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 10 monclient: shutdown 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 -- 10.20.193.51:0/2404594734 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 msgr2=0x7f70c00733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 --2- 10.20.193.51:0/2404594734 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c00733d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f70c006cf40 tx=0x7f70b002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 -- 10.20.193.51:0/2404594734 shutdown_connections 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 --2- 10.20.193.51:0/2404594734 >> v2:10.20.193.68:3300/0 conn(0x7f70c015d3b0 0x7f70c015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 --2- 10.20.193.51:0/2404594734 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c00733d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 -- 10.20.193.51:0/2404594734 >> 10.20.193.51:0/2404594734 conn(0x7f70c00ce350 msgr2=0x7f70c00ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 -- 10.20.193.51:0/2404594734 shutdown_connections 2026-02-22T21:37:41.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.535+0000 7f70c7a57640 1 -- 10.20.193.51:0/2404594734 wait complete. 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 10 monclient: monmap: 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 librados: starting msgr at 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 librados: starting objecter 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 Processor -- start 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 -- start start 2026-02-22T21:37:41.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 librados: setting wanted keys 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 1 librados: calling monclient init 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.536+0000 7f70c7a57640 10 monclient: init 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-c con 0x7f70c01526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f70c0152a80 0x7f70c0146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-b con 0x7f70c0152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f70c015d3b0 0x7f70c015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): picked mon.noname-a con 0x7f70c015d3b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f70c00748e0 con 0x7f70c015d3b0 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f70c0074600 con 0x7f70c0152a80 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f70c00cdcf0 con 0x7f70c01526a0 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c7a57640 10 monclient(hunting): authenticate will time out at 1137.117363s 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44202/0 (socket says 10.20.193.51:44202) 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 1 -- 10.20.193.51:0/3147640354 learned_addr learned my addr 10.20.193.51:0/3147640354 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 10 monclient(hunting): get_auth_request con 0x7f70c01526a0 auth_method 0 2026-02-22T21:37:41.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.537+0000 7f70c57cc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.538+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.538+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.538+0000 7f70c57cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc38801e470 con 0x7fc39415c4c0 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 10 monclient(hunting): handle_auth_done global_id 4547 payload 824 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 -- 10.20.193.51:0/3147640354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f70c0152a80 msgr2=0x7f70c0146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f70c0152a80 0x7f70c0146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 -- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f70c015d3b0 msgr2=0x7f70c015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f70c015d3b0 0x7f70c015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 monclient: found mon.noname-c 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7f70c57cc640 1 -- 10.20.193.51:0/3147640354 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c00c8c70 con 0x7f70c01526a0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7fc36ffff640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 0x7fc36407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7fc3937fe640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 0x7fc36407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.539+0000 7fc3937fe640 10 monclient: get_auth_request con 0x7fc364078e50 auth_method 0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.540+0000 7fc3937fe640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 0x7fc36407b310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc38000ac30 tx=0x7fc380008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.540+0000 7fc36ffff640 10 client.4573.objecter ms_handle_connect 0x7fc364078e50 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.540+0000 7f70c57cc640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70b002f3e0 tx=0x7f70b00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.540+0000 7f70adffb640 10 client.?.objecter ms_handle_connect 0x7f70c01526a0 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.540+0000 7f70adffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.541+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc38805db90 con 0x7fc39415c4c0 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.541+0000 7fc36ffff640 10 client.4573.objecter ms_dispatch2 0x7fc394000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.541+0000 7fc36ffff640 3 client.4573.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.541+0000 7fc36ffff640 3 client.4573.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.541+0000 7fc36ffff640 20 client.4573.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b003c040 con 0x7f70c01526a0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: dump: 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b003f680 con 0x7f70c01526a0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b003fa50 con 0x7f70c01526a0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70adffb640 10 monclient: dump: 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 5 monclient: authenticate success, global_id 4547 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70af7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _renew_subs 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c00c8f50 con 0x7f70c01526a0 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _renew_subs 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c00c9510 con 0x7f70c01526a0 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 1 librados: init done 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.542+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c0160630 con 0x7f70c01526a0 2026-02-22T21:37:41.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.553+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f70b003fbf0 con 0x7f70c01526a0 2026-02-22T21:37:41.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.554+0000 7f70adffb640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 0x7f709007b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.554+0000 7f70c4fcb640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 0x7f709007b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.554+0000 7f70c4fcb640 10 monclient: get_auth_request con 0x7f7090078e30 auth_method 0 2026-02-22T21:37:41.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.554+0000 7f70c4fcb640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 0x7f709007b2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f70b4007fd0 tx=0x7f70b4007b40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.554+0000 7f70adffb640 10 client.4547.objecter ms_handle_connect 0x7f7090078e30 2026-02-22T21:37:41.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.556+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f70b0083ad0 con 0x7f70c01526a0 2026-02-22T21:37:41.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.556+0000 7f70adffb640 10 client.4547.objecter ms_dispatch2 0x7f70c0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.556+0000 7f70adffb640 3 client.4547.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.556+0000 7f70adffb640 3 client.4547.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.556+0000 7f70adffb640 20 client.4547.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.586+0000 7f725af85640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.586+0000 7f725af85640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.586+0000 7f725af85640 10 monclient: monmap: 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.587386+0000 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.587386+0000 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.588 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 Processor -- start 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 -- start start 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient: init 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient: _add_conns ranks=[2,5,3,4,0,1] 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7254150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f725415c710 0x7f72540c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): picked mon.noname-c con 0x7f725415c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7254150600 0x7f725406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): picked mon.noname-a con 0x7f7254150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725415a920 con 0x7f7254150220 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f725415af60 con 0x7f7254150600 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f725415b240 con 0x7f725415c710 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.587+0000 7f725af85640 10 monclient(hunting): authenticate will time out at 1137.167812s 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f936402fa70 con 0x7f937015f710 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 10 client.4575.objecter ms_dispatch2 0x7f9370000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 3 client.4575.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 3 client.4575.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 20 client.4575.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9364086860 con 0x7f937015f710 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.596+0000 7f935dffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.604+0000 7f9f2e088640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.604+0000 7f9f2e088640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.604+0000 7f9f2e088640 10 monclient: monmap: 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.605579+0000 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.605579+0000 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.606 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.605+0000 7f9f2e088640 1 Processor -- start 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.605+0000 7f9f2e088640 1 -- start start 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.605+0000 7f9f2e088640 10 monclient: init 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient: _add_conns ranks=[3,4,0,1,2,5] 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9f2815f480 0x7f9f2815f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-a con 0x7f9f2815f480 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9f2815d110 0x7f9f280c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-b con 0x7f9f2815d110 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f280713b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f280c9ee0 con 0x7f9f280713b0 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f9f280c9510 con 0x7f9f2815f480 2026-02-22T21:37:41.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9f280c8620 con 0x7f9f2815d110 2026-02-22T21:37:41.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.606+0000 7f9f2e088640 10 monclient(hunting): authenticate will time out at 1137.186271s 2026-02-22T21:37:41.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.608+0000 7f72594fb640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7254150600 0x7f725406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f7254150600 0x7f725406fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33266/0 (socket says 10.20.193.51:33266) 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 1 -- 10.20.193.51:0/2270747243 learned_addr learned my addr 10.20.193.51:0/2270747243 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 10 monclient(hunting): get_auth_request con 0x7f7254150600 auth_method 0 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f72594fb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f9f27fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f9f280713b0 0x7f9f280736a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47102/0 (socket says 10.20.193.51:47102) 2026-02-22T21:37:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.610+0000 7f9f27fff640 1 -- 10.20.193.51:0/2045114817 learned_addr learned my addr 10.20.193.51:0/2045114817 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.611+0000 7f72594fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.611+0000 7f72594fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.611+0000 7f72594fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.613 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2026-02-22T21:37:41.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 10 monclient(hunting): handle_auth_done global_id 4578 payload 274 2026-02-22T21:37:41.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 1 -- 10.20.193.51:0/2270747243 >> v2:10.20.193.125:3300/0 conn(0x7f725415c710 msgr2=0x7f72540c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 1 --2- 10.20.193.51:0/2270747243 >> v2:10.20.193.125:3300/0 conn(0x7f725415c710 0x7f72540c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 1 -- 10.20.193.51:0/2270747243 >> v1:10.20.193.125:6789/0 conn(0x7f7254150220 legacy=0x7f725415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.612+0000 7f72594fb640 1 -- 10.20.193.51:0/2270747243 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725415b520 con 0x7f7254150600 2026-02-22T21:37:41.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f72594fb640 1 --2- 10.20.193.51:0/2270747243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f725406fca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7250009e50 tx=0x7f725002f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90409cd0c35600 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f9f277fe640 1 --2- 10.20.193.51:0/2045114817 >> v2:10.20.193.51:3300/0 conn(0x7f9f2815f480 0x7f9f2815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f9f277fe640 10 monclient(hunting): get_auth_request con 0x7f9f2815f480 auth_method 0 2026-02-22T21:37:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f9f277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f9f277fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.615+0000 7f9f277fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.625+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.625+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.625+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 10 monclient(hunting): handle_auth_done global_id 4584 payload 274 2026-02-22T21:37:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 1 -- 10.20.193.51:0/2045114817 >> v2:10.20.193.68:3300/0 conn(0x7f9f2815d110 msgr2=0x7f9f280c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 1 --2- 10.20.193.51:0/2045114817 >> v2:10.20.193.68:3300/0 conn(0x7f9f2815d110 0x7f9f280c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 1 -- 10.20.193.51:0/2045114817 >> v1:10.20.193.51:6789/0 conn(0x7f9f280713b0 legacy=0x7f9f280736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.626+0000 7f9f277fe640 1 -- 10.20.193.51:0/2045114817 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f280c8900 con 0x7f9f2815f480 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 1 -- 10.20.193.51:0/2270747243 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725002fd50 con 0x7f7254150600 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: dump: 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.633 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 1 -- 10.20.193.51:0/2270747243 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72500373f0 con 0x7f7254150600 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 1 -- 10.20.193.51:0/2270747243 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72500377c0 con 0x7f7254150600 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f725af85640 5 monclient: authenticate success, global_id 4578 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f725af85640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f725af85640 4 set_mon_vals no callback set 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f724b7fe640 10 monclient: dump: 2026-02-22T21:37:41.636 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.637 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.638 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.638 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.630+0000 7f9f277fe640 1 --2- 10.20.193.51:0/2045114817 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f2815f480 0x7f9f2815f860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f280721f0 tx=0x7f9f1402ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d64af8c44f06663 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 10 monclient: shutdown 2026-02-22T21:37:41.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 -- 10.20.193.51:0/2270747243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 msgr2=0x7f725406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 --2- 10.20.193.51:0/2270747243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f725406fca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7250009e50 tx=0x7f725002f330 comp rx=0 tx=0).stop 2026-02-22T21:37:41.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 -- 10.20.193.51:0/2270747243 shutdown_connections 2026-02-22T21:37:41.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 --2- 10.20.193.51:0/2270747243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f725406fca0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 --2- 10.20.193.51:0/2270747243 >> v2:10.20.193.125:3300/0 conn(0x7f725415c710 0x7f72540c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 -- 10.20.193.51:0/2270747243 >> 10.20.193.51:0/2270747243 conn(0x7f72540cd7b0 msgr2=0x7f72540cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 -- 10.20.193.51:0/2270747243 shutdown_connections 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 -- 10.20.193.51:0/2270747243 wait complete. 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 10 monclient: monmap: 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.644 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.645 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 librados: starting msgr at 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 librados: starting objecter 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.631+0000 7f725af85640 1 Processor -- start 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 -- start start 2026-02-22T21:37:41.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 librados: setting wanted keys 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 librados: calling monclient init 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient: init 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7254150220 0x7f7254072400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): picked mon.noname-b con 0x7f7254150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): picked mon.noname-a con 0x7f7254150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f725415c710 0x7f725414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): picked mon.noname-c con 0x7f725415c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f72540701e0 con 0x7f7254150600 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f725415bae0 con 0x7f7254150220 2026-02-22T21:37:41.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7254070360 con 0x7f725415c710 2026-02-22T21:37:41.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.632+0000 7f725af85640 10 monclient(hunting): authenticate will time out at 1137.212682s 2026-02-22T21:37:41.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 1 -- 10.20.193.51:0/2045114817 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1403c070 con 0x7f9f2815f480 2026-02-22T21:37:41.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: dump: 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.651 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 1 -- 10.20.193.51:0/2045114817 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f1402fa60 con 0x7f9f2815f480 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 1 -- 10.20.193.51:0/2045114817 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1402fe30 con 0x7f9f2815f480 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f2e088640 5 monclient: authenticate success, global_id 4584 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f2e088640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f2e088640 4 set_mon_vals no callback set 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f9f267fc640 10 monclient: dump: 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.653 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.654 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.633+0000 7f724bfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 10 monclient: shutdown 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33288/0 (socket says 10.20.193.51:33288) 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 1 -- 10.20.193.51:0/1319510998 learned_addr learned my addr 10.20.193.51:0/1319510998 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): get_auth_request con 0x7f7254150600 auth_method 0 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- 10.20.193.51:0/2045114817 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f2815f480 msgr2=0x7f9f2815f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 --2- 10.20.193.51:0/2045114817 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f2815f480 0x7f9f2815f860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f280721f0 tx=0x7f9f1402ed50 comp rx=0 tx=0).stop 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- 10.20.193.51:0/2045114817 shutdown_connections 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 --2- 10.20.193.51:0/2045114817 >> v2:10.20.193.68:3300/0 conn(0x7f9f2815d110 0x7f9f280c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 --2- 10.20.193.51:0/2045114817 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f2815f480 0x7f9f2815f860 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- 10.20.193.51:0/2045114817 >> 10.20.193.51:0/2045114817 conn(0x7f9f280ca680 msgr2=0x7f9f280caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- 10.20.193.51:0/2045114817 shutdown_connections 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- 10.20.193.51:0/2045114817 wait complete. 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f724bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 10 monclient: monmap: 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 librados: starting msgr at 2026-02-22T21:37:41.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 librados: starting objecter 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 Processor -- start 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 -- start start 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 librados: setting wanted keys 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 1 librados: calling monclient init 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.634+0000 7f9f2e088640 10 monclient: init 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-a con 0x7f9f280713b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9f2815d110 0x7f9f2814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-c con 0x7f9f2815d110 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9f2815f480 0x7f9f28150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): picked mon.noname-b con 0x7f9f2815f480 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9f280c91a0 con 0x7f9f280713b0 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9f280c8ec0 con 0x7f9f2815f480 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9f28076250 con 0x7f9f2815d110 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.635+0000 7f9f2e088640 10 monclient(hunting): authenticate will time out at 1137.215357s 2026-02-22T21:37:41.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 -- 10.20.193.51:0/1319510998 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f725415c710 msgr2=0x7f725414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f725415c710 0x7f725414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 -- 10.20.193.51:0/1319510998 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7254150220 msgr2=0x7f7254072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7254150220 0x7f7254072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.636+0000 7f724bfff640 1 -- 10.20.193.51:0/1319510998 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725406dc80 con 0x7f7254150600 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f724bfff640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f723c00b500 tx=0x7f723c00b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f72497fa640 10 client.?.objecter ms_handle_connect 0x7f7254150600 2026-02-22T21:37:41.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f72497fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.664 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2026-02-22T21:37:41.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33302/0 (socket says 10.20.193.51:33302) 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 1 -- 10.20.193.51:0/4157279583 learned_addr learned my addr 10.20.193.51:0/4157279583 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 10 monclient(hunting): get_auth_request con 0x7f9f280713b0 auth_method 0 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.637+0000 7f9f277fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.638+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.638+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.638+0000 7f9f277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 10 monclient(hunting): handle_auth_done global_id 4590 payload 824 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 -- 10.20.193.51:0/4157279583 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9f2815d110 msgr2=0x7f9f2814fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9f2815d110 0x7f9f2814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 -- 10.20.193.51:0/4157279583 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9f2815f480 msgr2=0x7f9f28150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9f2815f480 0x7f9f28150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.640+0000 7f9f277fe640 1 -- 10.20.193.51:0/4157279583 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f281f3ae0 con 0x7f9f280713b0 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c004280 con 0x7f7254150600 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: dump: 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.666 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f723c015070 con 0x7f7254150600 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 5 monclient: authenticate success, global_id 4587 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 10 monclient: _renew_subs 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725406df60 con 0x7f7254150600 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c0026e0 con 0x7f7254150600 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f724affd640 4 set_mon_vals no callback set 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f72497fa640 10 monclient: dump: 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.667 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 10 monclient: _renew_subs 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725406e520 con 0x7f7254150600 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.641+0000 7f725af85640 1 librados: init done 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f725af85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f725af85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f725af85640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f9f277fe640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9f1402f280 tx=0x7f9f140047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f9f03fff640 10 client.?.objecter ms_handle_connect 0x7f9f280713b0 2026-02-22T21:37:41.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725406fca0 con 0x7f7254150600 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.642+0000 7f9f03fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.645+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc38801a060 con 0x7fc39415c4c0 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.645+0000 7fc36ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.645+0000 7fc36ffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.659+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f70c0160630 con 0x7f70c01526a0 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.659+0000 7f70adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.659+0000 7f70adffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.680+0000 7f93773e3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.680+0000 7f93773e3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.680+0000 7f93773e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.680+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f9370076620 con 0x7f937015f710 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 10 monclient: monmap: 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.682 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.681743+0000 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.681743+0000 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 1 Processor -- start 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 1 -- start start 2026-02-22T21:37:41.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.681+0000 7f1484372640 10 monclient: init 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient: _add_conns ranks=[3,2,4,1,5,0] 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f147c06ff10 0x7f147c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): picked mon.noname-a con 0x7f147c06ff10 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f147c15a0f0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f147c153750 0x7f147c153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): picked mon.noname-b con 0x7f147c153750 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147c15d150 con 0x7f147c15a0f0 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f147c15d3f0 con 0x7f147c06ff10 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f147c15d6d0 con 0x7f147c153750 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f1484372640 10 monclient(hunting): authenticate will time out at 1137.262322s 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14828e8640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f147c153750 0x7f147c153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14818e6640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f147c15a0f0 0x7f147c15a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33320/0 (socket says 10.20.193.51:33320) 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14818e6640 1 -- 10.20.193.51:0/2965374011 learned_addr learned my addr 10.20.193.51:0/2965374011 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14828e8640 10 monclient(hunting): get_auth_request con 0x7f147c153750 auth_method 0 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14828e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14828e8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.682+0000 7f14828e8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.683+0000 7f14828e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.683+0000 7f14828e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.683+0000 7f14828e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 10 monclient(hunting): handle_auth_done global_id 4576 payload 274 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 1 -- 10.20.193.51:0/2965374011 >> v2:10.20.193.51:3300/0 conn(0x7f147c06ff10 msgr2=0x7f147c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 1 --2- 10.20.193.51:0/2965374011 >> v2:10.20.193.51:3300/0 conn(0x7f147c06ff10 0x7f147c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 1 -- 10.20.193.51:0/2965374011 >> v1:10.20.193.125:6789/0 conn(0x7f147c15a0f0 legacy=0x7f147c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.684+0000 7f14828e8640 1 -- 10.20.193.51:0/2965374011 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c15d9b0 con 0x7f147c153750 2026-02-22T21:37:41.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.685+0000 7f14828e8640 1 --2- 10.20.193.51:0/2965374011 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c153750 0x7f147c153ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f147c0c57d0 tx=0x7f147802f270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a0e979d7feac2b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.686+0000 7f14810e5640 1 -- 10.20.193.51:0/2965374011 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147803c070 con 0x7f147c153750 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: dump: 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 1 -- 10.20.193.51:0/2965374011 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1478037440 con 0x7f147c153750 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 1 -- 10.20.193.51:0/2965374011 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478037810 con 0x7f147c153750 2026-02-22T21:37:41.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f14810e5640 10 monclient: dump: 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 5 monclient: authenticate success, global_id 4576 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 4 set_mon_vals no callback set 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 10 monclient: shutdown 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 -- 10.20.193.51:0/2965374011 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c153750 msgr2=0x7f147c153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 --2- 10.20.193.51:0/2965374011 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c153750 0x7f147c153ba0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f147c0c57d0 tx=0x7f147802f270 comp rx=0 tx=0).stop 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 -- 10.20.193.51:0/2965374011 shutdown_connections 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 --2- 10.20.193.51:0/2965374011 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c153750 0x7f147c153ba0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 --2- 10.20.193.51:0/2965374011 >> v2:10.20.193.51:3300/0 conn(0x7f147c06ff10 0x7f147c0702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 -- 10.20.193.51:0/2965374011 >> 10.20.193.51:0/2965374011 conn(0x7f147c0cd7b0 msgr2=0x7f147c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 -- 10.20.193.51:0/2965374011 shutdown_connections 2026-02-22T21:37:41.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 1 -- 10.20.193.51:0/2965374011 wait complete. 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.687+0000 7f1484372640 10 monclient: monmap: 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.692 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 librados: starting msgr at 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 librados: starting objecter 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 Processor -- start 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 -- start start 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 librados: setting wanted keys 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 librados: calling monclient init 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient: init 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): picked mon.noname-b con 0x7f147c06ff10 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f147c153750 0x7f147c1f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): picked mon.noname-c con 0x7f147c153750 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f147c15a0f0 0x7f147c14b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): picked mon.noname-a con 0x7f147c15a0f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f147c159bd0 con 0x7f147c15a0f0 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f147c15df70 con 0x7f147c06ff10 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f147c0c9180 con 0x7f147c153750 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.688+0000 7f1484372640 10 monclient(hunting): authenticate will time out at 1137.268823s 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36026/0 (socket says 10.20.193.51:36026) 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 1 -- 10.20.193.51:0/2439566747 learned_addr learned my addr 10.20.193.51:0/2439566747 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): get_auth_request con 0x7f147c06ff10 auth_method 0 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14820e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.689+0000 7f14818e6640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f147c153750 0x7f147c1f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): get_auth_request con 0x7f147c153750 auth_method 0 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14818e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14820e7640 10 monclient(hunting): handle_auth_done global_id 4579 payload 824 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14820e7640 1 -- 10.20.193.51:0/2439566747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f147c153750 msgr2=0x7f147c1f7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.690+0000 7f14820e7640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f147c153750 0x7f147c1f7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 1 -- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f147c15a0f0 msgr2=0x7f147c14b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f147c15a0f0 0x7f147c14b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.691+0000 7f14820e7640 1 -- 10.20.193.51:0/2439566747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c0bf120 con 0x7f147c06ff10 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.692+0000 7f14820e7640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f146c00b780 tx=0x7f146c00bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.692+0000 7f1462ffd640 10 client.?.objecter ms_handle_connect 0x7f147c06ff10 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.692+0000 7f1462ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c004480 con 0x7f147c06ff10 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: dump: 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146c010070 con 0x7f147c06ff10 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c002920 con 0x7f147c06ff10 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1462ffd640 10 monclient: dump: 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.696 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f14808e4640 4 set_mon_vals no callback set 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 5 monclient: authenticate success, global_id 4579 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 10 monclient: _renew_subs 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147c0c0100 con 0x7f147c06ff10 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 10 monclient: _renew_subs 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147c1f8020 con 0x7f147c06ff10 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.693+0000 7f1484372640 1 librados: init done 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.694+0000 7f1484372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.694+0000 7f1484372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.694+0000 7f1484372640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.694+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f147c0c53d0 con 0x7f147c06ff10 2026-02-22T21:37:41.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.730+0000 7fc399cf5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:41.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.730+0000 7fc399cf5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:41.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.730+0000 7fc399cf5640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.730+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc394150ef0 con 0x7fc39415c4c0 2026-02-22T21:37:41.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.735+0000 7f6e19353640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.735+0000 7f6e19353640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.735+0000 7f6e19353640 10 monclient: monmap: 2026-02-22T21:37:41.736 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.736303+0000 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.736303+0000 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 Processor -- start 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 -- start start 2026-02-22T21:37:41.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient: init 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient: _add_conns ranks=[5,0,1,3,2,4] 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6e14150220 0x7f6e140c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-c con 0x7f6e14150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e14150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f6e1415c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e1415a920 con 0x7f6e14150600 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e1415af60 con 0x7f6e1415c710 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1415b240 con 0x7f6e14150220 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.736+0000 7f6e19353640 10 monclient(hunting): authenticate will time out at 1137.316886s 2026-02-22T21:37:41.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6e14150220 0x7f6e140c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6e14150220 0x7f6e140c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44258/0 (socket says 10.20.193.51:44258) 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 1 -- 10.20.193.51:0/4247528549 learned_addr learned my addr 10.20.193.51:0/4247528549 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): get_auth_request con 0x7f6e14150220 auth_method 0 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.737+0000 7f6e12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 10 monclient(hunting): handle_auth_done global_id 4553 payload 274 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 1 -- 10.20.193.51:0/4247528549 >> v1:10.20.193.68:6789/0 conn(0x7f6e1415c710 legacy=0x7f6e1406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 1 -- 10.20.193.51:0/4247528549 >> v1:10.20.193.51:6789/0 conn(0x7f6e14150600 legacy=0x7f6e1415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 1 monclient: found mon.noname-c 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.738+0000 7f6e12ffd640 1 -- 10.20.193.51:0/4247528549 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1415b520 con 0x7f6e14150220 2026-02-22T21:37:41.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.739+0000 7f6e12ffd640 1 --2- 10.20.193.51:0/4247528549 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150220 0x7f6e140c73a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009970 tx=0x7f6e0802efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb97ca65668420 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 1 -- 10.20.193.51:0/4247528549 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0803c070 con 0x7f6e14150220 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: dump: 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.742 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 5 monclient: authenticate success, global_id 4553 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 1 -- 10.20.193.51:0/4247528549 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e0802fbe0 con 0x7f6e14150220 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 1 -- 10.20.193.51:0/4247528549 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e08037440 con 0x7f6e14150220 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e11ffb640 10 monclient: dump: 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 4 set_mon_vals no callback set 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 10 monclient: shutdown 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 -- 10.20.193.51:0/4247528549 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150220 msgr2=0x7f6e140c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 --2- 10.20.193.51:0/4247528549 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150220 0x7f6e140c73a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009970 tx=0x7f6e0802efb0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 -- 10.20.193.51:0/4247528549 shutdown_connections 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 --2- 10.20.193.51:0/4247528549 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150220 0x7f6e140c73a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 -- 10.20.193.51:0/4247528549 >> 10.20.193.51:0/4247528549 conn(0x7f6e140cd7b0 msgr2=0x7f6e140cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 -- 10.20.193.51:0/4247528549 shutdown_connections 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.741+0000 7f6e19353640 1 -- 10.20.193.51:0/4247528549 wait complete. 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient: monmap: 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 librados: starting msgr at 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 librados: starting objecter 2026-02-22T21:37:41.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 Processor -- start 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 -- start start 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 librados: setting wanted keys 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 librados: calling monclient init 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient: init 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6e14150220 0x7f6e14159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-a con 0x7f6e14150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-c con 0x7f6e14150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6e1415c710 0x7f6e140cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1415c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6e140703b0 con 0x7f6e14150220 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1415bae0 con 0x7f6e1415c710 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6e14070530 con 0x7f6e14150600 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.742+0000 7f6e19353640 10 monclient(hunting): authenticate will time out at 1137.322956s 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44262/0 (socket says 10.20.193.51:44262) 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 1 -- 10.20.193.51:0/3672329811 learned_addr learned my addr 10.20.193.51:0/3672329811 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): get_auth_request con 0x7f6e14150600 auth_method 0 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.743+0000 7f6e127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.747 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 10 monclient(hunting): handle_auth_done global_id 4556 payload 824 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 -- 10.20.193.51:0/3672329811 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6e1415c710 msgr2=0x7f6e140cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6e1415c710 0x7f6e140cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 -- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6e14150220 msgr2=0x7f6e14159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6e14150220 0x7f6e14159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 monclient: found mon.noname-c 2026-02-22T21:37:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f6e127fc640 1 -- 10.20.193.51:0/3672329811 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e141f4520 con 0x7f6e14150600 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f70c7a57640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f70c7a57640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f70c7a57640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.745+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f70c015b780 con 0x7f70c01526a0 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.746+0000 7f6e127fc640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00b570 tx=0x7f6dfc00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.746+0000 7f6df37fe640 10 client.?.objecter ms_handle_connect 0x7f6e14150600 2026-02-22T21:37:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.746+0000 7f6df37fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.750 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.748+0000 7f70adffb640 1 -- 10.20.193.51:0/3147640354 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f70b0088190 con 0x7f70c01526a0 2026-02-22T21:37:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.748+0000 7f70adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2026-02-22T21:37:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.748+0000 7f70adffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 10 librados: watch_flush enter 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 10 librados: watch_flush exit 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 20 client.4547.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 10 client.4547.objecter successfully canceled tick 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 msgr2=0x7f709007b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 0x7f709007b2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f70b4007fd0 tx=0x7f70b4007b40 comp rx=0 tx=0).stop 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 10 monclient: shutdown 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 msgr2=0x7f70c0159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70b002f3e0 tx=0x7f70b00047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 shutdown_connections 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7090078e30 0x7f709007b2f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f70c015d3b0 0x7f70c015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f70c0152a80 0x7f70c0146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 --2- 10.20.193.51:0/3147640354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f70c01526a0 0x7f70c0159ee0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 >> 10.20.193.51:0/3147640354 conn(0x7f70c00ce350 msgr2=0x7f70c00ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 shutdown_connections 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 -- 10.20.193.51:0/3147640354 wait complete. 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc013020 con 0x7f6e14150600 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: dump: 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.752 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0044a0 con 0x7f6e14150600 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 5 monclient: authenticate success, global_id 4556 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _renew_subs 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e141f4800 con 0x7f6e14150600 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f70c7a57640 1 librados: shutdown 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0029c0 con 0x7f6e14150600 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e117fa640 4 set_mon_vals no callback set 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6df37fe640 10 monclient: dump: 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _renew_subs 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1414bf00 con 0x7f6e14150600 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 1 librados: init done 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.750+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e14074da0 con 0x7f6e14150600 2026-02-22T21:37:41.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f723c024430 con 0x7f7254150600 2026-02-22T21:37:41.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f72497fa640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 0x7f722407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f7258cfa640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 0x7f722407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f7258cfa640 10 monclient: get_auth_request con 0x7f7224078e50 auth_method 0 2026-02-22T21:37:41.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.0 2026-02-22T21:37:41.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f7258cfa640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 0x7f722407b310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7244006fd0 tx=0x7f7244006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.758+0000 7f72497fa640 10 client.4587.objecter ms_handle_connect 0x7f7224078e50 2026-02-22T21:37:41.760 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14046020 con 0x7f9f280713b0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: dump: 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f140426e0 con 0x7f9f280713b0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1403c040 con 0x7f9f280713b0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 5 monclient: authenticate success, global_id 4590 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 10 monclient: _renew_subs 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f25ffb640 4 set_mon_vals no callback set 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f281f3dc0 con 0x7f9f280713b0 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f03fff640 10 monclient: dump: 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 10 monclient: _renew_subs 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f281f4380 con 0x7f9f280713b0 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.761+0000 7f9f2e088640 1 librados: init done 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f9f2e088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f9f2e088640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f9f2e088640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6dfc002b60 con 0x7f6e14150600 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f28076630 con 0x7f9f280713b0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f6df37fe640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 0x7f6de007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f6e12ffd640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 0x7f6de007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.762+0000 7f6e12ffd640 10 monclient: get_auth_request con 0x7f6de0078e80 auth_method 0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.763+0000 7f6df37fe640 10 client.4556.objecter ms_handle_connect 0x7f6de0078e80 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.763+0000 7f6e12ffd640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 0x7f6de007b340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e14074940 tx=0x7f6e08031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.764+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f6dfc05e070 con 0x7f6e14150600 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.764+0000 7f6df37fe640 10 client.4556.objecter ms_dispatch2 0x7f6e14000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.764+0000 7f6df37fe640 3 client.4556.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.764+0000 7f6df37fe640 3 client.4556.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.764+0000 7f6df37fe640 20 client.4556.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f723c05dde0 con 0x7f7254150600 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 10 client.4587.objecter ms_dispatch2 0x7f7254000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 3 client.4587.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 3 client.4587.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 20 client.4587.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f723c017310 con 0x7f7254150600 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.767+0000 7f72497fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.771+0000 7f935dffb640 1 -- 10.20.193.51:0/2781007233 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f936408ab60 con 0x7f937015f710 2026-02-22T21:37:41.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.771+0000 7f935dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 8}] 2026-02-22T21:37:41.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.771+0000 7f935dffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.772 INFO:teuthology.orchestra.run.trial051.stdout:103079215109 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 10 librados: watch_flush enter 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 10 librados: watch_flush exit 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 20 client.4575.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 10 client.4575.objecter successfully canceled tick 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 msgr2=0x7f934007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 0x7f934007b310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9360007b40 tx=0x7f9360007a90 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 10 monclient: shutdown 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 msgr2=0x7f93700cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9364009130 tx=0x7f93640047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 shutdown_connections 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9340078e50 0x7f934007b310 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f937015faf0 0x7f9370151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f937015f710 0x7f93700cdaf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 --2- 10.20.193.51:0/2781007233 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f937015d8e0 0x7f937015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 >> 10.20.193.51:0/2781007233 conn(0x7f93700cf6c0 msgr2=0x7f93700cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 shutdown_connections 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 -- 10.20.193.51:0/2781007233 wait complete. 2026-02-22T21:37:41.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.773+0000 7f93773e3640 1 librados: shutdown 2026-02-22T21:37:41.778 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2026-02-22T21:37:41.780 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T21:37:41.780 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2026-02-22T21:37:41.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.8 2026-02-22T21:37:41.783 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.801+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f146c002ac0 con 0x7f147c06ff10 2026-02-22T21:37:41.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.802+0000 7f1462ffd640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 0x7f144c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.802+0000 7f14818e6640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 0x7f144c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.802+0000 7f14818e6640 10 monclient: get_auth_request con 0x7f144c078e00 auth_method 0 2026-02-22T21:37:41.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.802+0000 7f14818e6640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 0x7f144c07b2c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f147c14be70 tx=0x7f1470009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.802+0000 7f1462ffd640 10 client.4579.objecter ms_handle_connect 0x7f144c078e00 2026-02-22T21:37:41.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.803+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f146c05d5d0 con 0x7f147c06ff10 2026-02-22T21:37:41.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.803+0000 7f1462ffd640 10 client.4579.objecter ms_dispatch2 0x7f147c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.803+0000 7f1462ffd640 3 client.4579.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.803+0000 7f1462ffd640 3 client.4579.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.803+0000 7f1462ffd640 20 client.4579.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.809+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f147c0c53d0 con 0x7f147c06ff10 2026-02-22T21:37:41.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.809+0000 7f1462ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.809+0000 7f1462ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 10 monclient: monmap: 2026-02-22T21:37:41.820 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.820 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.820059+0000 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.820059+0000 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.821 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 1 Processor -- start 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 1 -- start start 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.819+0000 7f6d2a8a6640 10 monclient: init 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient: _add_conns ranks=[0,3,1,4,2,5] 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d24150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6d2415c710 0x7f6d240c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d2415c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f6d24150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d2415a920 con 0x7f6d24150220 2026-02-22T21:37:41.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d2415af60 con 0x7f6d24150600 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f6d2415b240 con 0x7f6d2415c710 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.820+0000 7f6d2a8a6640 10 monclient(hunting): authenticate will time out at 1137.400689s 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.821+0000 7f6d28e1c640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f6d24150600 0x7f6d2406fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54496/0 (socket says 10.20.193.51:54496) 2026-02-22T21:37:41.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.821+0000 7f6d28e1c640 1 -- 10.20.193.51:0/3541175625 learned_addr learned my addr 10.20.193.51:0/3541175625 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.846+0000 7fca034f5640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.846+0000 7fca034f5640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.846+0000 7fca034f5640 10 monclient: monmap: 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.847627+0000 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.847627+0000 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.848 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 10 monclient: monmap: 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.847860+0000 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.847860+0000 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.849 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 10 monclient: monmap: 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.847964+0000 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.847964+0000 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.850 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fca034f5640 1 Processor -- start 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fca034f5640 1 -- start start 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fca034f5640 10 monclient: init 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 1 Processor -- start 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 1 -- start start 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fe9e902c640 10 monclient: init 2026-02-22T21:37:41.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 1 Processor -- start 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 1 -- start start 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.847+0000 7fa965103640 10 monclient: init 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient: _add_conns ranks=[5,3,4,0,2,1] 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fc9fc1526a0 0x7fc9fc0733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc9fc1526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc9fc152a80 0x7fc9fc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc9fc0cdb20 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc9fc0cca00 con 0x7fc9fc152a80 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc9fc074c50 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fc9fc073d60 con 0x7fc9fc1526a0 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fca034f5640 10 monclient(hunting): authenticate will time out at 1137.428084s 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient: _add_conns ranks=[4,3,5,2,1,0] 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe9e41526a0 0x7fe9e40733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e41526a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe9e4152a80 0x7fe9e40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e4152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe9e40cdb20 0x7fe9e415d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e40cdb20 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e40cca00 con 0x7fe9e4152a80 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e4074c50 con 0x7fe9e41526a0 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e4073d60 con 0x7fe9e40cdb20 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fe9e902c640 10 monclient(hunting): authenticate will time out at 1137.428390s 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient: _add_conns ranks=[5,2,1,3,0,4] 2026-02-22T21:37:41.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa960155e60 0x7fa96015c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): picked mon.noname-c con 0x7fa960155e60 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9600cb570 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fa9600c9470 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9600cc450 con 0x7fa9600c9470 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9600cc730 con 0x7fa9600cb570 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fa96015ccd0 con 0x7fa960155e60 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.848+0000 7fa965103640 10 monclient(hunting): authenticate will time out at 1137.428601s 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.851+0000 7fc36ffff640 1 -- 10.20.193.51:0/2152880240 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc388062250 con 0x7fc39415c4c0 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.851+0000 7fc36ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.851+0000 7fc36ffff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.855 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fe9e2d76640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe9e41526a0 0x7fe9e40733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fca01a6b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 10 librados: watch_flush enter 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 10 librados: watch_flush exit 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 20 client.4573.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 10 client.4573.objecter successfully canceled tick 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 msgr2=0x7fc36407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 0x7fc36407b310 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc38000ac30 tx=0x7fc380008040 comp rx=0 tx=0).stop 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 10 monclient: shutdown 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 msgr2=0x7fc3940c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.853+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3940cb660 tx=0x7fc38800cf10 comp rx=0 tx=0).stop 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032960293 0 0) 0x7f6d2415af60 con 0x7f6d24150600 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 shutdown_connections 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc364078e50 0x7fc36407b310 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc39415c4c0 0x7fc3940c94c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 10 monclient(hunting): _negotiate 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 10 monclient(hunting): my global_id is 4582 2026-02-22T21:37:41.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3941592c0 0x7fc3941f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 --2- 10.20.193.51:0/2152880240 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc378004900 0x7fc394159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 >> 10.20.193.51:0/2152880240 conn(0x7fc3940ce350 msgr2=0x7fc3940d09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe9e41526a0 0x7fe9e40733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36042/0 (socket says 10.20.193.51:36042) 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 1 -- 10.20.193.51:0/931619735 learned_addr learned my addr 10.20.193.51:0/931619735 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): get_auth_request con 0x7fe9e41526a0 auth_method 0 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d08003620 con 0x7f6d24150600 2026-02-22T21:37:41.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 shutdown_connections 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 -- 10.20.193.51:0/2152880240 wait complete. 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fc399cf5640 1 librados: shutdown 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fa95f577640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fa9600c9470 0x7fa9600d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:54498/0 (socket says 10.20.193.51:54498) 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fa95f577640 1 -- 10.20.193.51:0/1705257183 learned_addr learned my addr 10.20.193.51:0/1705257183 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36030/0 (socket says 10.20.193.51:36030) 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 1 -- 10.20.193.51:0/1312330673 learned_addr learned my addr 10.20.193.51:0/1312330673 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): get_auth_request con 0x7fc9fc0cdb20 auth_method 0 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fe9e2d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.860 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.854+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 10 monclient(hunting): handle_auth_done global_id 4585 payload 274 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 -- 10.20.193.51:0/931619735 >> v2:10.20.193.125:3300/0 conn(0x7fe9e40cdb20 msgr2=0x7fe9e415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 --2- 10.20.193.51:0/931619735 >> v2:10.20.193.125:3300/0 conn(0x7fe9e40cdb20 0x7fe9e415d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 -- 10.20.193.51:0/931619735 >> v2:10.20.193.51:3300/0 conn(0x7fe9e4152a80 msgr2=0x7fe9e40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 --2- 10.20.193.51:0/931619735 >> v2:10.20.193.51:3300/0 conn(0x7fe9e4152a80 0x7fe9e40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.855+0000 7fe9e2d76640 1 -- 10.20.193.51:0/931619735 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e4074040 con 0x7fe9e41526a0 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 10 monclient(hunting): handle_auth_done global_id 4588 payload 274 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 -- 10.20.193.51:0/1312330673 >> v2:10.20.193.125:3300/0 conn(0x7fc9fc1526a0 msgr2=0x7fc9fc0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1312330673 >> v2:10.20.193.125:3300/0 conn(0x7fc9fc1526a0 0x7fc9fc0733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 -- 10.20.193.51:0/1312330673 >> v2:10.20.193.51:3300/0 conn(0x7fc9fc152a80 msgr2=0x7fc9fc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1312330673 >> v2:10.20.193.51:3300/0 conn(0x7fc9fc152a80 0x7fc9fc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.856+0000 7fca01a6b640 1 -- 10.20.193.51:0/1312330673 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc074040 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.859+0000 7fe9e2d76640 1 --2- 10.20.193.51:0/931619735 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e40733d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9e406cf40 tx=0x7fe9d802ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8342266a4ac9aad2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.859+0000 7f725af85640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.859+0000 7f725af85640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.859+0000 7f725af85640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.859+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7254074da0 con 0x7f7254150600 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.860+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1312330673 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc08f0d0 tx=0x7fc9f802f2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc503c8efe328348 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004436563 0 0) 0x7fa9600cc450 con 0x7fa9600c9470 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 10 monclient(hunting): _negotiate 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 10 monclient(hunting): my global_id is 4591 2026-02-22T21:37:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.861+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa94c003620 con 0x7fa9600c9470 2026-02-22T21:37:41.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.6 2026-02-22T21:37:41.864 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3999351940 0 0) 0x7f6d08003620 con 0x7f6d24150600 2026-02-22T21:37:41.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 >> v2:10.20.193.51:3300/0 conn(0x7f6d2415c710 msgr2=0x7f6d240c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 --2- 10.20.193.51:0/3541175625 >> v2:10.20.193.51:3300/0 conn(0x7f6d2415c710 0x7f6d240c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.865 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 >> v1:10.20.193.51:6789/0 conn(0x7f6d24150220 legacy=0x7f6d2415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2415b520 con 0x7f6d24150600 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f6d18003280 con 0x7f6d24150600 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d2a8a6640 5 monclient: authenticate success, global_id 4582 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d2a8a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: dump: 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.866 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 >> v1:10.20.193.68:6789/0 conn(0x7f6d24150600 legacy=0x7f6d2406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08003ba0 0x7f6d08003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): picked mon.a con 0x7f6d08003ba0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 0x7f6d08004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): picked mon.b con 0x7f6d08004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d08005470 0x7f6d0800bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): picked mon.c con 0x7f6d08005470 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2415b240 con 0x7f6d08003ba0 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0800c3c0 con 0x7f6d08004900 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0800c5c0 con 0x7f6d08005470 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d22ffd640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:41.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.863+0000 7f6d2a8a6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6dfc01d020 con 0x7f6e14150600 2026-02-22T21:37:41.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7f6df37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7f6df37fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 1 -- 10.20.193.51:0/931619735 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d803c070 con 0x7fe9e41526a0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: dump: 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 1 -- 10.20.193.51:0/931619735 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d802fbd0 con 0x7fe9e41526a0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 1 -- 10.20.193.51:0/931619735 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8037440 con 0x7fe9e41526a0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e1d74640 10 monclient: dump: 2026-02-22T21:37:41.869 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 5 monclient: authenticate success, global_id 4585 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 4 set_mon_vals no callback set 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 10 monclient: shutdown 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 -- 10.20.193.51:0/931619735 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 msgr2=0x7fe9e40733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 --2- 10.20.193.51:0/931619735 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e40733d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9e406cf40 tx=0x7fe9d802ef50 comp rx=0 tx=0).stop 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 -- 10.20.193.51:0/931619735 shutdown_connections 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 --2- 10.20.193.51:0/931619735 >> v2:10.20.193.125:3300/0 conn(0x7fe9e40cdb20 0x7fe9e415d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 --2- 10.20.193.51:0/931619735 >> v2:10.20.193.51:3300/0 conn(0x7fe9e4152a80 0x7fe9e40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 --2- 10.20.193.51:0/931619735 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e40733d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 -- 10.20.193.51:0/931619735 >> 10.20.193.51:0/931619735 conn(0x7fe9e40ce350 msgr2=0x7fe9e40ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 -- 10.20.193.51:0/931619735 shutdown_connections 2026-02-22T21:37:41.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fe9e902c640 1 -- 10.20.193.51:0/931619735 wait complete. 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 1 -- 10.20.193.51:0/1312330673 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f802fd80 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: dump: 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.871 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 1 -- 10.20.193.51:0/1312330673 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f80373f0 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fca034f5640 5 monclient: authenticate success, global_id 4588 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fca034f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 1 -- 10.20.193.51:0/1312330673 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80377c0 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fca034f5640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fca034f5640 4 set_mon_vals no callback set 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.864+0000 7fc9ebfff640 10 monclient: dump: 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.872 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient: monmap: 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 librados: starting msgr at 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 librados: starting objecter 2026-02-22T21:37:41.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 10 monclient: shutdown 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 Processor -- start 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 -- 10.20.193.51:0/1312330673 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 msgr2=0x7fc9fc15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 --2- 10.20.193.51:0/1312330673 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc08f0d0 tx=0x7fc9f802f2d0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 -- start start 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 librados: setting wanted keys 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 librados: calling monclient init 2026-02-22T21:37:41.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient: init 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 -- 10.20.193.51:0/1312330673 shutdown_connections 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 --2- 10.20.193.51:0/1312330673 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc15d3b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 --2- 10.20.193.51:0/1312330673 >> v2:10.20.193.51:3300/0 conn(0x7fc9fc152a80 0x7fc9fc0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 --2- 10.20.193.51:0/1312330673 >> v2:10.20.193.125:3300/0 conn(0x7fc9fc1526a0 0x7fc9fc0733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 -- 10.20.193.51:0/1312330673 >> 10.20.193.51:0/1312330673 conn(0x7fc9fc0ce350 msgr2=0x7fc9fc0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 -- 10.20.193.51:0/1312330673 shutdown_connections 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 -- 10.20.193.51:0/1312330673 wait complete. 2026-02-22T21:37:41.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe9e40cdb20 0x7fe9e41f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e40cdb20 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e41526a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe9e4152a80 0x7fe9e40c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e4152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e40748e0 con 0x7fe9e40cdb20 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4074600 con 0x7fe9e41526a0 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4073910 con 0x7fe9e4152a80 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fe9e902c640 10 monclient(hunting): authenticate will time out at 1137.445688s 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 10 monclient: monmap: 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.877 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 librados: starting msgr at 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 librados: starting objecter 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.865+0000 7fca034f5640 1 Processor -- start 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 -- start start 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 librados: setting wanted keys 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 librados: calling monclient init 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient: init 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc1f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc9fc0cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9fc1526a0 0x7fc9fc14c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc1526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc9fc152a80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc0748e0 con 0x7fc9fc1526a0 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc074600 con 0x7fc9fc0cdb20 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc073910 con 0x7fc9fc152a80 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fca034f5640 10 monclient(hunting): authenticate will time out at 1137.446594s 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3368233619 0 0) 0x7fa94c003620 con 0x7fa9600c9470 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 >> v2:10.20.193.125:3300/0 conn(0x7fa960155e60 msgr2=0x7fa96015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 --2- 10.20.193.51:0/1705257183 >> v2:10.20.193.125:3300/0 conn(0x7fa960155e60 0x7fa96015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 >> v1:10.20.193.125:6789/0 conn(0x7fa9600cb570 legacy=0x7fa9600c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9600cca10 con 0x7fa9600c9470 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa95dd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa965103640 5 monclient: authenticate success, global_id 4591 2026-02-22T21:37:41.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fa965103640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7f6d237fe640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 0x7f6d08004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.866+0000 7fe9e2575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): get_auth_request con 0x7f6d08004900 auth_method 0 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36054/0 (socket says 10.20.193.51:36054) 2026-02-22T21:37:41.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 1 -- 10.20.193.51:0/3160823745 learned_addr learned my addr 10.20.193.51:0/3160823745 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 10 monclient(hunting): get_auth_request con 0x7fe9e41526a0 auth_method 0 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7fe9e2575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.867+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7fe9e2575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7fe9e2575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7f6d28e1c640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d08005470 0x7f6d0800bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7fe9e2575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.882 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2026-02-22T21:37:41.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7f6d28e1c640 10 monclient(hunting): get_auth_request con 0x7f6d08005470 auth_method 0 2026-02-22T21:37:41.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7f6d28e1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7f6d28e1c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.868+0000 7f6d28e1c640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 10 monclient(hunting): handle_auth_done global_id 4582 payload 293 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 -- 10.20.193.51:0/3541175625 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d08005470 msgr2=0x7f6d0800bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d08005470 0x7f6d0800bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 -- 10.20.193.51:0/3541175625 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08003ba0 msgr2=0x7f6d08003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08003ba0 0x7f6d08003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 monclient: found mon.b 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d28e1c640 1 -- 10.20.193.51:0/3541175625 reap_dead start 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 1 -- 10.20.193.51:0/3541175625 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d08003620 con 0x7f6d08004900 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.869+0000 7f6d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e3577640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe9e4152a80 0x7fe9e40c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 -- 10.20.193.51:0/3160823745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe9e4152a80 msgr2=0x7fe9e40c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe9e4152a80 0x7fe9e40c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 -- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe9e40cdb20 msgr2=0x7fe9e41f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe9e40cdb20 0x7fe9e41f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 monclient: found mon.noname-b 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.870+0000 7fe9e2575640 1 -- 10.20.193.51:0/3160823745 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e41f8000 con 0x7fe9e41526a0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.872+0000 7f6d237fe640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 0x7f6d08004d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009980 tx=0x7f6d1402ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874c7973be438c36 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.872+0000 7fe9e2575640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00b780 tx=0x7fe9cc00bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.872+0000 7fe9c37fe640 10 client.?.objecter ms_handle_connect 0x7fe9e41526a0 2026-02-22T21:37:41.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.872+0000 7fe9c37fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.874+0000 7fca01a6b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44298/0 (socket says 10.20.193.51:44298) 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 1 -- 10.20.193.51:0/1475173637 learned_addr learned my addr 10.20.193.51:0/1475173637 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): get_auth_request con 0x7fc9fc152a80 auth_method 0 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fa950003280 con 0x7fa9600c9470 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:37:41.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient: dump: 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 >> v1:10.20.193.68:6789/0 conn(0x7fa9600c9470 legacy=0x7fa9600d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa94c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): picked mon.c con 0x7fa94c003b50 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa94c004900 0x7fa94c004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): picked mon.b con 0x7fa94c004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa94c0054c0 0x7fa94c00bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): picked mon.a con 0x7fa94c0054c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa96015ccd0 con 0x7fa94c0054c0 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c00c400 con 0x7fa94c004900 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c00c600 con 0x7fa94c003b50 2026-02-22T21:37:41.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa95dd74640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fa965103640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fca01a6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc004480 con 0x7fe9e41526a0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: dump: 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9cc010070 con 0x7fe9e41526a0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 5 monclient: authenticate success, global_id 4594 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 10 monclient: _renew_subs 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e41f7480 con 0x7fe9e41526a0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc002920 con 0x7fe9e41526a0 2026-02-22T21:37:41.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e1573640 4 set_mon_vals no callback set 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9c37fe640 10 monclient: dump: 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 10 monclient: _renew_subs 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e41f7a40 con 0x7fe9e41526a0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.875+0000 7fe9e902c640 1 librados: init done 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fe9e902c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fe9e902c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fe9e902c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e406a0b0 con 0x7fe9e41526a0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fca0126a640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc1f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.876+0000 7fa95e575640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa94c004900 0x7fa94c004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca0126a640 10 monclient(hunting): get_auth_request con 0x7fc9fc0cdb20 auth_method 0 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca0126a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca0126a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca0126a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fa95e575640 10 monclient(hunting): get_auth_request con 0x7fa94c004900 auth_method 0 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fa95e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fa95e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fa95e575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 10 monclient(hunting): handle_auth_done global_id 4559 payload 824 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 -- 10.20.193.51:0/1475173637 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 msgr2=0x7fc9fc1f4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc1f4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 -- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9fc1526a0 msgr2=0x7fc9fc14c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9fc1526a0 0x7fc9fc14c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 monclient: found mon.noname-c 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.877+0000 7fca01a6b640 1 -- 10.20.193.51:0/1475173637 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc14c7a0 con 0x7fc9fc152a80 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.878+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa94c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): get_auth_request con 0x7fa94c003b50 auth_method 0 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fca01a6b640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8039040 tx=0x7fc9f8004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fc9e9ffb640 10 client.?.objecter ms_handle_connect 0x7fc9fc152a80 2026-02-22T21:37:41.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fc9e9ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.879+0000 7fa95ed76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 10 monclient(hunting): handle_auth_done global_id 4591 payload 293 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 -- 10.20.193.51:0/1705257183 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa94c004900 msgr2=0x7fa94c004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa94c004900 0x7fa94c004d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 -- 10.20.193.51:0/1705257183 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa94c0054c0 msgr2=0x7fa94c00bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa94c0054c0 0x7fa94c00bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 monclient: found mon.c 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 1 -- 10.20.193.51:0/1705257183 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c003620 con 0x7fa94c003b50 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95ed76640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.881+0000 7fa95f577640 1 -- 10.20.193.51:0/1705257183 reap_dead start 2026-02-22T21:37:41.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.882+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa94c003f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa954009980 tx=0x7fa95402ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=966fc2242ca931e8 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.899+0000 7f1484372640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:41.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.899+0000 7f1484372640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:41.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.899+0000 7f1484372640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:41.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.899+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 15} v 0) -- 0x7f147c1519a0 con 0x7f147c06ff10 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8046020 con 0x7fc9fc152a80 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: dump: 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f8037e40 con 0x7fc9fc152a80 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8041670 con 0x7fc9fc152a80 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9e9ffb640 10 monclient: dump: 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fc9eb7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 5 monclient: authenticate success, global_id 4559 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 10 monclient: _renew_subs 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9fc14ca80 con 0x7fc9fc152a80 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 10 monclient: _renew_subs 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9fc14d040 con 0x7fc9fc152a80 2026-02-22T21:37:41.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.911+0000 7fca034f5640 1 librados: init done 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95403c070 con 0x7fa94c003b50 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: dump: 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa95402f7b0 con 0x7fa94c003b50 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 1 -- 10.20.193.51:0/1705257183 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954009550 con 0x7fa94c003b50 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa95dd74640 10 monclient: dump: 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.915 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 4 set_mon_vals no callback set 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fca034f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fca034f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fca034f5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9fc06a0b0 con 0x7fc9fc152a80 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 10 monclient: shutdown 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 -- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 msgr2=0x7fa94c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa94c003f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa954009980 tx=0x7fa95402ed50 comp rx=0 tx=0).stop 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 -- 10.20.193.51:0/1705257183 shutdown_connections 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 --2- 10.20.193.51:0/1705257183 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa94c003f50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 -- 10.20.193.51:0/1705257183 >> 10.20.193.51:0/1705257183 conn(0x7fa96006d8c0 msgr2=0x7fa96006dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 -- 10.20.193.51:0/1705257183 shutdown_connections 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.912+0000 7fa965103640 1 -- 10.20.193.51:0/1705257183 wait complete. 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 10 monclient: monmap: 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 librados: starting msgr at 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 librados: starting objecter 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 Processor -- start 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 -- start start 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 librados: setting wanted keys 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 1 librados: calling monclient init 2026-02-22T21:37:41.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.913+0000 7fa965103640 10 monclient: init 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa9601f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): picked mon.noname-c con 0x7fa94c003b50 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa96015f540 0x7fa9601f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): picked mon.noname-b con 0x7fa96015f540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): picked mon.noname-a con 0x7fa96015f9c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa9600ccfd0 con 0x7fa96015f9c0 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa9600cc200 con 0x7fa96015f540 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa9600cd2b0 con 0x7fa94c003b50 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.918 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.914+0000 7fa965103640 10 monclient(hunting): authenticate will time out at 1137.494055s 2026-02-22T21:37:41.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.936+0000 7f72497fa640 1 -- 10.20.193.51:0/1319510998 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f723c0624a0 con 0x7f7254150600 2026-02-22T21:37:41.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.936+0000 7f72497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2026-02-22T21:37:41.937 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.936+0000 7f72497fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:41.938 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:41.939 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 10 librados: watch_flush enter 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 10 librados: watch_flush exit 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 20 client.4587.objecter shutdown clearing up homeless session... 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 10 client.4587.objecter successfully canceled tick 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 msgr2=0x7f722407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 0x7f722407b310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7244006fd0 tx=0x7f7244006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 10 monclient: shutdown 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 msgr2=0x7f72540c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f723c00b500 tx=0x7f723c00b9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 shutdown_connections 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7224078e50 0x7f722407b310 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f725415c710 0x7f725414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7254150600 0x7f72540c9490 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 --2- 10.20.193.51:0/1319510998 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7254150220 0x7f7254072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 >> 10.20.193.51:0/1319510998 conn(0x7f72540cd7b0 msgr2=0x7f72540ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 shutdown_connections 2026-02-22T21:37:41.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 -- 10.20.193.51:0/1319510998 wait complete. 2026-02-22T21:37:41.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.938+0000 7f725af85640 1 librados: shutdown 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.944+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9f1403f030 con 0x7f9f280713b0 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.945+0000 7f9f03fff640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 0x7f9ef807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.945+0000 7f9f26ffd640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 0x7f9ef807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.945+0000 7f9f26ffd640 10 monclient: get_auth_request con 0x7f9ef8078e50 auth_method 0 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.945+0000 7f9f03fff640 10 client.4590.objecter ms_handle_connect 0x7f9ef8078e50 2026-02-22T21:37:41.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.945+0000 7f9f26ffd640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 0x7f9ef807b310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9f2806fd00 tx=0x7f9f18006d90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.2 2026-02-22T21:37:41.948 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:41.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.947+0000 7f6e19353640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:41.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.947+0000 7f6e19353640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:41.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.947+0000 7f6e19353640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:41.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.947+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 12} v 0) -- 0x7f6e1414f8e0 con 0x7f6e14150600 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.952+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9f140834a0 con 0x7f9f280713b0 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.952+0000 7f9f03fff640 10 client.4590.objecter ms_dispatch2 0x7f9f28000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.952+0000 7f9f03fff640 3 client.4590.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.952+0000 7f9f03fff640 3 client.4590.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.953+0000 7f9f03fff640 20 client.4590.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.953+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9f14050030 con 0x7f9f280713b0 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.953+0000 7f9f03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.953+0000 7f9f03fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.954+0000 7fa95f577640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33426/0 (socket says 10.20.193.51:33426) 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 1 -- 10.20.193.51:0/1304886727 learned_addr learned my addr 10.20.193.51:0/1304886727 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): get_auth_request con 0x7fa96015f9c0 auth_method 0 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.955+0000 7fa95f577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 10 monclient(hunting): handle_auth_done global_id 4593 payload 824 2026-02-22T21:37:41.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 -- 10.20.193.51:0/1304886727 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 msgr2=0x7fa9601f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa9601f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 -- 10.20.193.51:0/1304886727 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa96015f540 msgr2=0x7fa9601f5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa96015f540 0x7fa9601f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.957+0000 7fa95f577640 1 -- 10.20.193.51:0/1304886727 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9601f3c80 con 0x7fa96015f9c0 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 10 monclient: get_monmap_and_config 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 10 monclient: monmap: 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:41.959069+0000 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:41.959069+0000 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.959 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 1 Processor -- start 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 1 -- start start 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.958+0000 7f2a2c606640 10 monclient: init 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient: _add_conns ranks=[4,3,0,2,5,1] 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f2a24150220 0x7f2a240c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-b con 0x7f2a24150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f2a24150600 0x7f2a2415c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-a con 0x7f2a24150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a2415c540 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2415c9b0 con 0x7f2a2415c540 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2415a920 con 0x7f2a24150600 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2415af60 con 0x7f2a24150220 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.959+0000 7f2a2c606640 10 monclient(hunting): authenticate will time out at 1137.539433s 2026-02-22T21:37:41.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.968+0000 7fa95f577640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa9600c86a0 tx=0x7fa95000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.968+0000 7fa93f7fe640 10 client.?.objecter ms_handle_connect 0x7fa96015f9c0 2026-02-22T21:37:41.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.968+0000 7fa93f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.970+0000 7f2a29b7a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f2a24150600 0x7f2a2415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f2a24150600 0x7f2a2415c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33440/0 (socket says 10.20.193.51:33440) 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3953835217 learned_addr learned my addr 10.20.193.51:0/3953835217 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): get_auth_request con 0x7f2a24150600 auth_method 0 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950014420 con 0x7fa96015f9c0 2026-02-22T21:37:41.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: dump: 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9500145c0 con 0x7fa96015f9c0 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 5 monclient: authenticate success, global_id 4593 2026-02-22T21:37:41.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 10 monclient: _renew_subs 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9601f3f60 con 0x7fa96015f9c0 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9500149b0 con 0x7fa96015f9c0 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa95d573640 4 set_mon_vals no callback set 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa93f7fe640 10 monclient: dump: 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.975 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 10 monclient: _renew_subs 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9601f4b60 con 0x7fa96015f9c0 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7fa965103640 1 librados: init done 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.971+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:41.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.972+0000 7fa965103640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.972+0000 7fa965103640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.972+0000 7fa965103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.972+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9600cd590 con 0x7fa96015f9c0 2026-02-22T21:37:41.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_done global_id 4596 payload 274 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3953835217 >> v2:10.20.193.68:3300/0 conn(0x7f2a24150220 msgr2=0x7f2a240c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 1 --2- 10.20.193.51:0/3953835217 >> v2:10.20.193.68:3300/0 conn(0x7f2a24150220 0x7f2a240c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3953835217 >> v1:10.20.193.51:6789/0 conn(0x7f2a2415c540 legacy=0x7f2a2406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 10 monclient: _finish_hunting 0 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 1 monclient: found mon.noname-a 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:41.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.973+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3953835217 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2415b240 con 0x7f2a24150600 2026-02-22T21:37:41.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.982+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe9cc01d030 con 0x7fe9e41526a0 2026-02-22T21:37:41.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.983+0000 7fe9c37fe640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 0x7fe9b007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.983+0000 7fe9e2d76640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 0x7fe9b007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.983+0000 7fe9e2d76640 10 monclient: get_auth_request con 0x7fe9b0078e50 auth_method 0 2026-02-22T21:37:41.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.983+0000 7fe9c37fe640 10 client.4594.objecter ms_handle_connect 0x7fe9b0078e50 2026-02-22T21:37:41.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.983+0000 7fe9e2d76640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 0x7fe9b007b310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe9e415f410 tx=0x7fe9d8031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:41.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe9cc01de30 con 0x7fe9e41526a0 2026-02-22T21:37:41.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7fe9c37fe640 10 client.4594.objecter ms_dispatch2 0x7fe9e4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:41.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7fe9c37fe640 3 client.4594.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:41.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7fe9c37fe640 3 client.4594.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:41.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7fe9c37fe640 20 client.4594.objecter dump_active .. 0 homeless 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1403c070 con 0x7f6d08004900 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: dump: 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.988 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1402f790 con 0x7f6d08004900 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 1 -- 10.20.193.51:0/3541175625 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14009550 con 0x7f6d08004900 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 4 set_mon_vals no callback set 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d22ffd640 10 monclient: dump: 2026-02-22T21:37:41.989 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 10 monclient: shutdown 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 msgr2=0x7f6d08004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 0x7f6d08004d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009980 tx=0x7f6d1402ed30 comp rx=0 tx=0).stop 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/3541175625 shutdown_connections 2026-02-22T21:37:41.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/3541175625 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d08004900 0x7f6d08004d80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/3541175625 >> 10.20.193.51:0/3541175625 conn(0x7f6d240cd7b0 msgr2=0x7f6d240cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/3541175625 shutdown_connections 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.985+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/3541175625 wait complete. 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 10 monclient: build_initial_monmap 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 10 monclient: monmap: 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 librados: starting msgr at 2026-02-22T21:37:41.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 librados: starting objecter 2026-02-22T21:37:41.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 Processor -- start 2026-02-22T21:37:41.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 -- start start 2026-02-22T21:37:41.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 librados: setting wanted keys 2026-02-22T21:37:41.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 1 librados: calling monclient init 2026-02-22T21:37:41.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.986+0000 7f6d2a8a6640 10 monclient: init 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08004900 0x7f6d241f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d08004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d2414ce30 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d24157610 0x7f6d241ea830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d24157610 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6d240c73a0 con 0x7f6d08004900 2026-02-22T21:37:41.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2415bae0 con 0x7f6d2414ce30 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6d240c7520 con 0x7f6d24157610 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.987+0000 7f6d2a8a6640 10 monclient(hunting): authenticate will time out at 1137.567359s 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.991+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe9e406a0b0 con 0x7fe9e41526a0 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.991+0000 7fe9c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:41.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.991+0000 7fe9c37fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:41.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.996+0000 7f6d237fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:41.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36106/0 (socket says 10.20.193.51:36106) 2026-02-22T21:37:41.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 1 -- 10.20.193.51:0/2265384524 learned_addr learned my addr 10.20.193.51:0/2265384524 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:41.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 10 monclient(hunting): get_auth_request con 0x7f6d2414ce30 auth_method 0 2026-02-22T21:37:41.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:41.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:41.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.997+0000 7f6d237fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:41.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:41.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:41.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f6d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f1462ffd640 1 -- 10.20.193.51:0/2439566747 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 15}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f147c1519a0 con 0x7f147c06ff10 2026-02-22T21:37:42.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f1462ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 15}] 2026-02-22T21:37:42.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:41.998+0000 7f1462ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.000 INFO:teuthology.orchestra.run.trial051.stdout:90194313222 2026-02-22T21:37:42.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 -- 10.20.193.51:0/2265384524 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d24157610 msgr2=0x7f6d241ea830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d24157610 0x7f6d241ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 -- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08004900 msgr2=0x7f6d241f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08004900 0x7f6d241f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 monclient: found mon.noname-b 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f6d237fe640 1 -- 10.20.193.51:0/2265384524 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d241f47b0 con 0x7f6d2414ce30 2026-02-22T21:37:42.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f1484372640 10 librados: watch_flush enter 2026-02-22T21:37:42.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f1484372640 10 librados: watch_flush exit 2026-02-22T21:37:42.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f1484372640 20 client.4579.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.000+0000 7f1484372640 10 client.4579.objecter successfully canceled tick 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 msgr2=0x7f144c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 0x7f144c07b2c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f147c14be70 tx=0x7f1470009040 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 10 monclient: shutdown 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 msgr2=0x7f147c0bfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f146c00b780 tx=0x7f146c00bc50 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 shutdown_connections 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f144c078e00 0x7f144c07b2c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f147c15a0f0 0x7f147c14b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f147c153750 0x7f147c1f7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 --2- 10.20.193.51:0/2439566747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f147c06ff10 0x7f147c0bfbc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 >> 10.20.193.51:0/2439566747 conn(0x7f147c0cd7b0 msgr2=0x7f147c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 shutdown_connections 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 -- 10.20.193.51:0/2439566747 wait complete. 2026-02-22T21:37:42.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f1484372640 1 librados: shutdown 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f6d237fe640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6d1402ed70 tx=0x7f6d140048d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f6d20ff9640 10 client.?.objecter ms_handle_connect 0x7f6d2414ce30 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.001+0000 7f6d20ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14045020 con 0x7f6d2414ce30 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: dump: 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.006 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d14004070 con 0x7f6d2414ce30 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1403c040 con 0x7f6d2414ce30 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 5 monclient: authenticate success, global_id 4600 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 10 monclient: _renew_subs 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d227fc640 4 set_mon_vals no callback set 2026-02-22T21:37:42.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d24157b00 con 0x7f6d2414ce30 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d20ff9640 10 monclient: dump: 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 10 monclient: _renew_subs 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d241580c0 con 0x7f6d2414ce30 2026-02-22T21:37:42.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.003+0000 7f6d2a8a6640 1 librados: init done 2026-02-22T21:37:42.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.004+0000 7f6d2a8a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.004+0000 7f6d2a8a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.004+0000 7f6d2a8a6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.004+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2415c000 con 0x7f6d2414ce30 2026-02-22T21:37:42.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.005+0000 7f21b4beb640 10 monclient: get_monmap_and_config 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.005+0000 7f21b4beb640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.005+0000 7f21b4beb640 10 monclient: monmap: 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:42.006577+0000 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:42.006577+0000 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:42.011 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 Processor -- start 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 -- start start 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient: init 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient: _add_conns ranks=[2,0,3,4,1,5] 2026-02-22T21:37:42.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21b01526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21b0152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f21b015d3b0 0x7f21b015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-a con 0x7f21b015d3b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b0073ae0 con 0x7f21b0152a80 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b0073d60 con 0x7f21b01526a0 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f21b00cca00 con 0x7f21b015d3b0 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.006+0000 7f21b4beb640 10 monclient(hunting): authenticate will time out at 1137.586952s 2026-02-22T21:37:42.014 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313222 got 90194313222 for osd.15 2026-02-22T21:37:42.015 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.016+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc9f8041810 con 0x7fc9fc152a80 2026-02-22T21:37:42.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.016+0000 7fc9e9ffb640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 0x7fc9cc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.016+0000 7fca0126a640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 0x7fc9cc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.016+0000 7fca0126a640 10 monclient: get_auth_request con 0x7fc9cc078e50 auth_method 0 2026-02-22T21:37:42.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fca0126a640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 0x7fc9cc07b310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0098f0 tx=0x7fc9ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fc9e9ffb640 10 client.4559.objecter ms_handle_connect 0x7fc9cc078e50 2026-02-22T21:37:42.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc9f8083c30 con 0x7fc9fc152a80 2026-02-22T21:37:42.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fc9e9ffb640 10 client.4559.objecter ms_dispatch2 0x7fc9fc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fc9e9ffb640 3 client.4559.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.017+0000 7fc9e9ffb640 3 client.4559.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.018+0000 7fc9e9ffb640 20 client.4559.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.024+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc9fc06a0b0 con 0x7fc9fc152a80 2026-02-22T21:37:42.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.024+0000 7fc9e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.024+0000 7fc9e9ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.027+0000 7f6df37fe640 1 -- 10.20.193.51:0/3672329811 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 12}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f6dfc062730 con 0x7f6e14150600 2026-02-22T21:37:42.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.027+0000 7f6df37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 12}] 2026-02-22T21:37:42.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.027+0000 7f6df37fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.029 INFO:teuthology.orchestra.run.trial051.stdout:103079215109 2026-02-22T21:37:42.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f21ae575640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f21b01526a0 0x7f21b00733d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33348/0 (socket says 10.20.193.51:33348) 2026-02-22T21:37:42.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f21ae575640 1 -- 10.20.193.51:0/1926476747 learned_addr learned my addr 10.20.193.51:0/1926476747 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 10 librados: watch_flush enter 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 10 librados: watch_flush exit 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 20 client.4556.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 10 client.4556.objecter successfully canceled tick 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 msgr2=0x7f6de007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 0x7f6de007b340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e14074940 tx=0x7f6e08031000 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 10 monclient: shutdown 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 msgr2=0x7f6e1414c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.029+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00b570 tx=0x7f6dfc00ba40 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 shutdown_connections 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6de0078e80 0x7f6de007b340 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6e1415c710 0x7f6e140cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6e14150600 0x7f6e1414c700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 --2- 10.20.193.51:0/3672329811 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6e14150220 0x7f6e14159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 >> 10.20.193.51:0/3672329811 conn(0x7f6e140cd7b0 msgr2=0x7f6e1415c000 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 shutdown_connections 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 -- 10.20.193.51:0/3672329811 wait complete. 2026-02-22T21:37:42.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.030+0000 7f6e19353640 1 librados: shutdown 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546809451 0 0) 0x7f21b0073d60 con 0x7f21b01526a0 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 10 monclient(hunting): _negotiate 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 10 monclient(hunting): my global_id is 4562 2026-02-22T21:37:42.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.031+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f219c003620 con 0x7f21b01526a0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 343192591 0 0) 0x7f219c003620 con 0x7f21b01526a0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 >> v2:10.20.193.51:3300/0 conn(0x7f21b015d3b0 msgr2=0x7f21b015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 --2- 10.20.193.51:0/1926476747 >> v2:10.20.193.51:3300/0 conn(0x7f21b015d3b0 0x7f21b015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 >> v1:10.20.193.51:6789/0 conn(0x7f21b0152a80 legacy=0x7f21b00cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b0074040 con 0x7f21b01526a0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f21a4002f00 con 0x7f21b01526a0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: dump: 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.034 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 >> v1:10.20.193.125:6789/0 conn(0x7f21b01526a0 legacy=0x7f21b00733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f219c003ba0 0x7f219c003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): picked mon.a con 0x7f219c003ba0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 0x7f219c004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): picked mon.c con 0x7f219c004900 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c005470 0x7f219c00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): picked mon.b con 0x7f219c005470 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f21b00cca00 con 0x7f219c003ba0 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f219c00c3c0 con 0x7f219c005470 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f219c00c5c0 con 0x7f219c004900 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.033+0000 7f21ad573640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 0x7f219c004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): get_auth_request con 0x7f219c004900 auth_method 0 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:42.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.034+0000 7f21add74640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:42.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 10 monclient(hunting): handle_auth_done global_id 4562 payload 293 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 -- 10.20.193.51:0/1926476747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c005470 msgr2=0x7f219c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c005470 0x7f219c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 -- 10.20.193.51:0/1926476747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f219c003ba0 msgr2=0x7f219c003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f219c003ba0 0x7f219c003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 monclient: found mon.c 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 1 -- 10.20.193.51:0/1926476747 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219c003620 con 0x7f219c004900 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21add74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21aed76640 1 -- 10.20.193.51:0/1926476747 reap_dead start 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21b4beb640 5 monclient: authenticate success, global_id 4562 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.035+0000 7f21b4beb640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:42.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.036+0000 7f21add74640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 0x7f219c004d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2198002f60 tx=0x7f219802eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=748a56017a0160b8 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198002b30 con 0x7f219c004900 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: dump: 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2198005290 con 0x7f219c004900 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 1 -- 10.20.193.51:0/1926476747 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198003710 con 0x7f219c004900 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 4 set_mon_vals no callback set 2026-02-22T21:37:42.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21ad573640 10 monclient: dump: 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.040 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.12 2026-02-22T21:37:42.040 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 10 monclient: shutdown 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 -- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 msgr2=0x7f219c004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 0x7f219c004d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2198002f60 tx=0x7f219802eca0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 -- 10.20.193.51:0/1926476747 shutdown_connections 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 --2- 10.20.193.51:0/1926476747 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f219c004900 0x7f219c004d80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 -- 10.20.193.51:0/1926476747 >> 10.20.193.51:0/1926476747 conn(0x7f21b00ce350 msgr2=0x7f21b00ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 -- 10.20.193.51:0/1926476747 shutdown_connections 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.038+0000 7f21b4beb640 1 -- 10.20.193.51:0/1926476747 wait complete. 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient: monmap: 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:42.041 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 librados: starting msgr at 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 librados: starting objecter 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 Processor -- start 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 -- start start 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 librados: setting wanted keys 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 librados: calling monclient init 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient: init 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c004900 0x7f21b01f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-b con 0x7f219c004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f21b015c600 0x7f21b015e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-a con 0x7f21b015c600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): picked mon.noname-c con 0x7f21b015ca80 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0074600 con 0x7f21b015c600 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0074c50 con 0x7f219c004900 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f21b00748e0 con 0x7f21b015ca80 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f21b4beb640 10 monclient(hunting): authenticate will time out at 1137.619845s 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f9f2e088640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:42.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f9f2e088640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f9f2e088640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.039+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9f280c8010 con 0x7f9f280713b0 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44340/0 (socket says 10.20.193.51:44340) 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 1 -- 10.20.193.51:0/880110950 learned_addr learned my addr 10.20.193.51:0/880110950 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): get_auth_request con 0x7f21b015ca80 auth_method 0 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.040+0000 7f21aed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 10 monclient(hunting): handle_auth_done global_id 4565 payload 824 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 -- 10.20.193.51:0/880110950 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c004900 msgr2=0x7f21b01f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c004900 0x7f21b01f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 -- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f21b015c600 msgr2=0x7f21b015e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f21b015c600 0x7f21b015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 monclient: found mon.noname-c 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.042+0000 7f21aed76640 1 -- 10.20.193.51:0/880110950 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b015aaf0 con 0x7f21b015ca80 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.043+0000 7f21aed76640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f219c0058e0 tx=0x7f21a000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.043+0000 7f2196ffd640 10 client.?.objecter ms_handle_connect 0x7f21b015ca80 2026-02-22T21:37:42.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.043+0000 7f2196ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0017070 con 0x7f21b015ca80 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: dump: 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a000aa70 con 0x7f21b015ca80 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0007e60 con 0x7f21b015ca80 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f2196ffd640 10 monclient: dump: 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21acd72640 4 set_mon_vals no callback set 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 5 monclient: authenticate success, global_id 4565 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 10 monclient: _renew_subs 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b015db60 con 0x7f21b015ca80 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 10 monclient: _renew_subs 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b015e120 con 0x7f21b015ca80 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.044+0000 7f21b4beb640 1 librados: init done 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.045+0000 7f21b4beb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.045+0000 7f21b4beb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.045+0000 7f21b4beb640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.045+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b015d820 con 0x7f21b015ca80 2026-02-22T21:37:42.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f21a001b3b0 con 0x7f21b015ca80 2026-02-22T21:37:42.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f2196ffd640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 0x7f217c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f21ae575640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 0x7f217c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f21ae575640 10 monclient: get_auth_request con 0x7f217c078e00 auth_method 0 2026-02-22T21:37:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f2196ffd640 10 client.4565.objecter ms_handle_connect 0x7f217c078e00 2026-02-22T21:37:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.057+0000 7f21ae575640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 0x7f217c07b2c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21a40036e0 tx=0x7f21a4002e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.079+0000 7fe9e902c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:42.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.079+0000 7fe9e902c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:42.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.079+0000 7fe9e902c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.079+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe9e40733d0 con 0x7fe9e41526a0 2026-02-22T21:37:42.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.080+0000 7f2a29b7a640 1 --2- 10.20.193.51:0/3953835217 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a2415c000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a24070940 tx=0x7f2a1802ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b0a7ba56e8b783 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.087+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa9500133c0 con 0x7fa96015f9c0 2026-02-22T21:37:42.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.088+0000 7fa93f7fe640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 0x7fa92c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.088+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 0x7fa92c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.088+0000 7fa95ed76640 10 monclient: get_auth_request con 0x7fa92c078e80 auth_method 0 2026-02-22T21:37:42.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.088+0000 7fa93f7fe640 10 client.4593.objecter ms_handle_connect 0x7fa92c078e80 2026-02-22T21:37:42.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.088+0000 7fa95ed76640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 0x7fa92c07b340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa954039000 tx=0x7fa954031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa950013d70 con 0x7fa96015f9c0 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 10 client.4593.objecter ms_dispatch2 0x7fa960000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 3 client.4593.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 3 client.4593.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 20 client.4593.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa9600cd590 con 0x7fa96015f9c0 2026-02-22T21:37:42.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.096+0000 7fa93f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 1 -- 10.20.193.51:0/3953835217 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1803c070 con 0x7f2a24150600 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: dump: 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.106 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 1 -- 10.20.193.51:0/3953835217 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a1802fbd0 con 0x7f2a24150600 2026-02-22T21:37:42.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 1 -- 10.20.193.51:0/3953835217 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18037440 con 0x7f2a24150600 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a2c606640 5 monclient: authenticate success, global_id 4596 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a2c606640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a2c606640 4 set_mon_vals no callback set 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.104+0000 7f2a29379640 10 monclient: dump: 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.108 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.109 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.109 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 10 monclient: shutdown 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 -- 10.20.193.51:0/3953835217 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 msgr2=0x7f2a2415c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3953835217 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a2415c000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a24070940 tx=0x7f2a1802ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 -- 10.20.193.51:0/3953835217 shutdown_connections 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3953835217 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a2415c000 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3953835217 >> v2:10.20.193.68:3300/0 conn(0x7f2a24150220 0x7f2a240c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 -- 10.20.193.51:0/3953835217 >> 10.20.193.51:0/3953835217 conn(0x7f2a240cd7b0 msgr2=0x7f2a240cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 -- 10.20.193.51:0/3953835217 shutdown_connections 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.105+0000 7f2a2c606640 1 -- 10.20.193.51:0/3953835217 wait complete. 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 10 monclient: monmap: 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 librados: starting msgr at 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 librados: starting objecter 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 Processor -- start 2026-02-22T21:37:42.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 -- start start 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 librados: setting wanted keys 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 1 librados: calling monclient init 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 10 monclient: init 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.106+0000 7f2a2c606640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2a24150220 0x7f2a241f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-c con 0x7f2a24150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-a con 0x7f2a24150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2a2415c540 0x7f2a24158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2415c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2415bae0 con 0x7f2a24150600 2026-02-22T21:37:42.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2415b800 con 0x7f2a2415c540 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2a240703b0 con 0x7f2a24150220 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.107+0000 7f2a2c606640 10 monclient(hunting): authenticate will time out at 1137.687049s 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.108+0000 7f2a29b7a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.108+0000 7f9f03fff640 1 -- 10.20.193.51:0/4157279583 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f140363a0 con 0x7f9f280713b0 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.108+0000 7f9f03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2026-02-22T21:37:42.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.108+0000 7f9f03fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.115 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:42.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33462/0 (socket says 10.20.193.51:33462) 2026-02-22T21:37:42.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3679228509 learned_addr learned my addr 10.20.193.51:0/3679228509 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:42.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): get_auth_request con 0x7f2a24150600 auth_method 0 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.109+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6d140377f0 con 0x7f6d2414ce30 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f6d20ff9640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 0x7f6cf407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f6d23fff640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 0x7f6cf407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f6d23fff640 10 monclient: get_auth_request con 0x7f6cf4078e50 auth_method 0 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f6d23fff640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 0x7f6cf407b310 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6d2406c780 tx=0x7f6d10006d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f6d20ff9640 10 client.4600.objecter ms_handle_connect 0x7f6cf4078e50 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3679228509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2a24150220 msgr2=0x7f2a241f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2a24150220 0x7f2a241f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3679228509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2a2415c540 msgr2=0x7f2a24158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2a2415c540 0x7f2a24158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 monclient: found mon.noname-a 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f9f2e088640 10 librados: watch_flush enter 2026-02-22T21:37:42.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f2a29b7a640 1 -- 10.20.193.51:0/3679228509 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a241f48a0 con 0x7f2a24150600 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f9f2e088640 10 librados: watch_flush exit 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f9f2e088640 20 client.4590.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.110+0000 7f9f2e088640 10 client.4590.objecter successfully canceled tick 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 msgr2=0x7f9ef807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 0x7f9ef807b310 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9f2806fd00 tx=0x7f9f18006d90 comp rx=0 tx=0).stop 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 10 monclient: shutdown 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 msgr2=0x7f9f28070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9f1402f280 tx=0x7f9f140047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 shutdown_connections 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ef8078e50 0x7f9ef807b310 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9f2815f480 0x7f9f28150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9f2815d110 0x7f9f2814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 --2- 10.20.193.51:0/4157279583 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9f280713b0 0x7f9f28070240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 >> 10.20.193.51:0/4157279583 conn(0x7f9f280ca680 msgr2=0x7f9f281e71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 shutdown_connections 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 -- 10.20.193.51:0/4157279583 wait complete. 2026-02-22T21:37:42.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f9f2e088640 1 librados: shutdown 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f6d14004440 con 0x7f6d2414ce30 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f6d20ff9640 10 client.4600.objecter ms_dispatch2 0x7f6d24000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f6d20ff9640 3 client.4600.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f6d20ff9640 3 client.4600.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.111+0000 7f6d20ff9640 20 client.4600.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.112+0000 7f2a29b7a640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a18002410 tx=0x7f2a180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.112+0000 7f2a0affd640 10 client.?.objecter ms_handle_connect 0x7f2a24150600 2026-02-22T21:37:42.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.112+0000 7f2a0affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:42.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.113+0000 7fca034f5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:42.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.113+0000 7fca034f5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:42.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.113+0000 7fca034f5640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.113+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 14} v 0) -- 0x7fc9fc0733d0 con 0x7fc9fc152a80 2026-02-22T21:37:42.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18044070 con 0x7f2a24150600 2026-02-22T21:37:42.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: dump: 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a18040630 con 0x7f2a24150600 2026-02-22T21:37:42.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1803c040 con 0x7f2a24150600 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a28b78640 4 set_mon_vals no callback set 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a0affd640 10 monclient: dump: 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 5 monclient: authenticate success, global_id 4599 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 10 monclient: _renew_subs 2026-02-22T21:37:42.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a24157770 con 0x7f2a24150600 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 10 monclient: _renew_subs 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a24157d30 con 0x7f2a24150600 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.114+0000 7f2a2c606640 1 librados: init done 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.115+0000 7f2a2c606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.115+0000 7f2a2c606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.115+0000 7f2a2c606640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.115+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2415c000 con 0x7f2a24150600 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.117+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6d1404f030 con 0x7f6d2414ce30 2026-02-22T21:37:42.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.117+0000 7f6d20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.117+0000 7f6d20ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.3 2026-02-22T21:37:42.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.122+0000 7fe9c37fe640 1 -- 10.20.193.51:0/3160823745 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe9cc024080 con 0x7fe9e41526a0 2026-02-22T21:37:42.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.122+0000 7fe9c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2026-02-22T21:37:42.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.122+0000 7fe9c37fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.129 INFO:teuthology.orchestra.run.trial051.stdout:103079215108 2026-02-22T21:37:42.129 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 10 librados: watch_flush enter 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 10 librados: watch_flush exit 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 20 client.4594.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 10 client.4594.objecter successfully canceled tick 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 msgr2=0x7fe9b007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 0x7fe9b007b310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe9e415f410 tx=0x7fe9d8031000 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 10 monclient: shutdown 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 msgr2=0x7fe9e41f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00b780 tx=0x7fe9cc00bc50 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 shutdown_connections 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe9b0078e50 0x7fe9b007b310 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe9e4152a80 0x7fe9e40c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe9e41526a0 0x7fe9e41f4750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 --2- 10.20.193.51:0/3160823745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe9e40cdb20 0x7fe9e41f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 >> 10.20.193.51:0/3160823745 conn(0x7fe9e40ce350 msgr2=0x7fe9e40ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 shutdown_connections 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 -- 10.20.193.51:0/3160823745 wait complete. 2026-02-22T21:37:42.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.124+0000 7fe9e902c640 1 librados: shutdown 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.126+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f2a1803f050 con 0x7f2a24150600 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.127+0000 7f2a0affd640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 0x7f29f407b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.127+0000 7f2a2a37b640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 0x7f29f407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.127+0000 7f2a2a37b640 10 monclient: get_auth_request con 0x7f29f4078e80 auth_method 0 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.127+0000 7f2a0affd640 10 client.4599.objecter ms_handle_connect 0x7f29f4078e80 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.127+0000 7f2a2a37b640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 0x7f29f407b340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098a0 tx=0x7f2a14006d90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.128+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f2a18082ff0 con 0x7f2a24150600 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.128+0000 7f2a0affd640 10 client.4599.objecter ms_dispatch2 0x7f2a24000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.128+0000 7f2a0affd640 3 client.4599.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.128+0000 7f2a0affd640 3 client.4599.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.128+0000 7f2a0affd640 20 client.4599.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.134 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.1 2026-02-22T21:37:42.134 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.150+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f21a005d5d0 con 0x7f21b015ca80 2026-02-22T21:37:42.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.150+0000 7f2196ffd640 10 client.4565.objecter ms_dispatch2 0x7f21b0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.150+0000 7f2196ffd640 3 client.4565.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.150+0000 7f2196ffd640 3 client.4565.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.150+0000 7f2196ffd640 20 client.4565.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.157+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f21a0021020 con 0x7f21b015ca80 2026-02-22T21:37:42.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.157+0000 7f2196ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.157+0000 7f2196ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.160+0000 7fc9e9ffb640 1 -- 10.20.193.51:0/1475173637 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 14}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7fc9f8050030 con 0x7fc9fc152a80 2026-02-22T21:37:42.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.160+0000 7fc9e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 14}] 2026-02-22T21:37:42.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.160+0000 7fc9e9ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.161 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 10 librados: watch_flush enter 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 10 librados: watch_flush exit 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 20 client.4559.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 10 client.4559.objecter successfully canceled tick 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 msgr2=0x7fc9cc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 0x7fc9cc07b310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0098f0 tx=0x7fc9ec008040 comp rx=0 tx=0).stop 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 10 monclient: shutdown 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 msgr2=0x7fc9fc0d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8039040 tx=0x7fc9f8004b40 comp rx=0 tx=0).stop 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 shutdown_connections 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc9cc078e50 0x7fc9cc07b310 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc9fc152a80 0x7fc9fc0d0b30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc9fc1526a0 0x7fc9fc14c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 --2- 10.20.193.51:0/1475173637 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc9fc0cdb20 0x7fc9fc1f4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 >> 10.20.193.51:0/1475173637 conn(0x7fc9fc0ce350 msgr2=0x7fc9fc14ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 shutdown_connections 2026-02-22T21:37:42.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 -- 10.20.193.51:0/1475173637 wait complete. 2026-02-22T21:37:42.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.162+0000 7fca034f5640 1 librados: shutdown 2026-02-22T21:37:42.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.14 2026-02-22T21:37:42.172 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.184+0000 7fa965103640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:42.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.184+0000 7fa965103640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:42.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.184+0000 7fa965103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.184+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fa96006e6e0 con 0x7fa96015f9c0 2026-02-22T21:37:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.204+0000 7f6d2a8a6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.204+0000 7f6d2a8a6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.204+0000 7f6d2a8a6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.204+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f6d24074da0 con 0x7f6d2414ce30 2026-02-22T21:37:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.206+0000 7f6d20ff9640 1 -- 10.20.193.51:0/2265384524 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6d241f47b0 con 0x7f6d2414ce30 2026-02-22T21:37:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.206+0000 7f6d20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 9}] 2026-02-22T21:37:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.206+0000 7f6d20ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.208 INFO:teuthology.orchestra.run.trial051.stdout:98784247812 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 10 librados: watch_flush enter 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 10 librados: watch_flush exit 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 20 client.4600.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 10 client.4600.objecter successfully canceled tick 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 msgr2=0x7f6cf407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 0x7f6cf407b310 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6d2406c780 tx=0x7f6d10006d40 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 10 monclient: shutdown 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 msgr2=0x7f6d241ea2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6d1402ed70 tx=0x7f6d140048d0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 shutdown_connections 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cf4078e50 0x7f6cf407b310 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d24157610 0x7f6d241ea830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d2414ce30 0x7f6d241ea2f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 --2- 10.20.193.51:0/2265384524 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d08004900 0x7f6d241f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 >> 10.20.193.51:0/2265384524 conn(0x7f6d240cd7b0 msgr2=0x7f6d240cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 shutdown_connections 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 -- 10.20.193.51:0/2265384524 wait complete. 2026-02-22T21:37:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.208+0000 7f6d2a8a6640 1 librados: shutdown 2026-02-22T21:37:42.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247812 for osd.9 2026-02-22T21:37:42.218 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.233+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f2a180876b0 con 0x7f2a24150600 2026-02-22T21:37:42.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.233+0000 7f2a0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.233+0000 7f2a0affd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.241+0000 7fa93f7fe640 1 -- 10.20.193.51:0/1304886727 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7fa95001a060 con 0x7fa96015f9c0 2026-02-22T21:37:42.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.241+0000 7fa93f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 10}] 2026-02-22T21:37:42.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.241+0000 7fa93f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.243 INFO:teuthology.orchestra.run.trial051.stdout:98784247813 2026-02-22T21:37:42.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.242+0000 7f21b4beb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:42.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.242+0000 7f21b4beb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:42.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.242+0000 7f21b4beb640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:37:42.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.242+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f21b00c9990 con 0x7f21b015ca80 2026-02-22T21:37:42.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 10 librados: watch_flush enter 2026-02-22T21:37:42.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 10 librados: watch_flush exit 2026-02-22T21:37:42.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 20 client.4593.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 10 client.4593.objecter successfully canceled tick 2026-02-22T21:37:42.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 msgr2=0x7fa92c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 0x7fa92c07b340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa954039000 tx=0x7fa954031000 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 10 monclient: shutdown 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 msgr2=0x7fa9601f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa9600c86a0 tx=0x7fa95000a430 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 shutdown_connections 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa92c078e80 0x7fa92c07b340 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa96015f9c0 0x7fa9601f58a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa96015f540 0x7fa9601f5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 --2- 10.20.193.51:0/1304886727 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa94c003b50 0x7fa9601f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 >> 10.20.193.51:0/1304886727 conn(0x7fa96006d8c0 msgr2=0x7fa960153680 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 shutdown_connections 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 -- 10.20.193.51:0/1304886727 wait complete. 2026-02-22T21:37:42.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.243+0000 7fa965103640 1 librados: shutdown 2026-02-22T21:37:42.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.244+0000 7f2196ffd640 1 -- 10.20.193.51:0/880110950 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f21a000ae60 con 0x7f21b015ca80 2026-02-22T21:37:42.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.244+0000 7f2196ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 11}] 2026-02-22T21:37:42.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.244+0000 7f2196ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.247 INFO:teuthology.orchestra.run.trial051.stdout:94489280517 2026-02-22T21:37:42.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 10 librados: watch_flush enter 2026-02-22T21:37:42.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 10 librados: watch_flush exit 2026-02-22T21:37:42.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 20 client.4565.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 10 client.4565.objecter successfully canceled tick 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 msgr2=0x7f217c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 0x7f217c07b2c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f21a40036e0 tx=0x7f21a4002e80 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 10 monclient: shutdown 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 msgr2=0x7f21b015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f219c0058e0 tx=0x7f21a000cce0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 shutdown_connections 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f217c078e00 0x7f217c07b2c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f21b015ca80 0x7f21b015ee60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f21b015c600 0x7f21b015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 --2- 10.20.193.51:0/880110950 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f219c004900 0x7f21b01f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 >> 10.20.193.51:0/880110950 conn(0x7f21b00ce350 msgr2=0x7f21b00ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.246+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 shutdown_connections 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.247+0000 7f21b4beb640 1 -- 10.20.193.51:0/880110950 wait complete. 2026-02-22T21:37:42.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.247+0000 7f21b4beb640 1 librados: shutdown 2026-02-22T21:37:42.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247813 for osd.10 2026-02-22T21:37:42.254 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.11 2026-02-22T21:37:42.256 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.316+0000 7f2a2c606640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:42.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.316+0000 7f2a2c606640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:42.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.317+0000 7f2a2c606640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.317+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 13} v 0) -- 0x7f2a24074da0 con 0x7f2a24150600 2026-02-22T21:37:42.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.319+0000 7f2a0affd640 1 -- 10.20.193.51:0/3679228509 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 13}]=0 v0) ==== 75+0+12 (secure 0 0 0) 0x7f2a18086fb0 con 0x7f2a24150600 2026-02-22T21:37:42.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.319+0000 7f2a0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 13}] 2026-02-22T21:37:42.320 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.319+0000 7f2a0affd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:42.321 INFO:teuthology.orchestra.run.trial051.stdout:98784247814 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 10 librados: watch_flush enter 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 10 librados: watch_flush exit 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 20 client.4599.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 10 client.4599.objecter successfully canceled tick 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 msgr2=0x7f29f407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 0x7f29f407b340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098a0 tx=0x7f2a14006d90 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 10 monclient: shutdown 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 msgr2=0x7f2a240c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a18002410 tx=0x7f2a180047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 shutdown_connections 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f29f4078e80 0x7f29f407b340 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2a2415c540 0x7f2a24158530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2a24150600 0x7f2a240c8ab0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 --2- 10.20.193.51:0/3679228509 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2a24150220 0x7f2a241f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.320+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 >> 10.20.193.51:0/3679228509 conn(0x7f2a240cd7b0 msgr2=0x7f2a240cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.321+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 shutdown_connections 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.321+0000 7f2a2c606640 1 -- 10.20.193.51:0/3679228509 wait complete. 2026-02-22T21:37:42.322 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.321+0000 7f2a2c606640 1 librados: shutdown 2026-02-22T21:37:42.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247814 for osd.13 2026-02-22T21:37:42.331 DEBUG:teuthology.parallel:result is None 2026-02-22T21:37:42.331 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T21:37:42.331 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:37:42.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.395+0000 7fb6d8206640 10 monclient: get_monmap_and_config 2026-02-22T21:37:42.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.395+0000 7fb6d8206640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.395+0000 7fb6d8206640 10 monclient: monmap: 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:42.396399+0000 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:42.396399+0000 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:42.398 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 1 Processor -- start 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 1 -- start start 2026-02-22T21:37:42.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient: init 2026-02-22T21:37:42.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient: _add_conns ranks=[2,0,1,5,3,4] 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6d0150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6d015c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb6d00703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d015a920 con 0x7fb6d015c710 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d015af60 con 0x7fb6d00703b0 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d015b240 con 0x7fb6d0150220 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.396+0000 7fb6d8206640 10 monclient(hunting): authenticate will time out at 1137.976703s 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.397+0000 7fb6d577a640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fb6d015c710 0x7fb6d006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47146/0 (socket says 10.20.193.51:47146) 2026-02-22T21:37:42.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.397+0000 7fb6d577a640 1 -- 10.20.193.51:0/755240990 learned_addr learned my addr 10.20.193.51:0/755240990 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435701972 0 0) 0x7fb6d015a920 con 0x7fb6d015c710 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _negotiate 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): my global_id is 4602 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b8003620 con 0x7fb6d015c710 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 425032018 0 0) 0x7fb6d015af60 con 0x7fb6d00703b0 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _negotiate 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): my global_id is 4603 2026-02-22T21:37:42.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d015a920 con 0x7fb6d00703b0 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047876006 0 0) 0x7fb6d015b240 con 0x7fb6d0150220 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _negotiate 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 10 monclient(hunting): my global_id is 4568 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.398+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d015af60 con 0x7fb6d0150220 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2885758549 0 0) 0x7fb6b8003620 con 0x7fb6d015c710 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 >> v1:10.20.193.125:6789/0 conn(0x7fb6d0150220 legacy=0x7fb6d015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 >> v1:10.20.193.68:6789/0 conn(0x7fb6d00703b0 legacy=0x7fb6d0159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:37:42.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d015b520 con 0x7fb6d015c710 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fb6c4002f00 con 0x7fb6d015c710 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.399+0000 7fb6d4f79640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient: dump: 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.408 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 >> v1:10.20.193.51:6789/0 conn(0x7fb6d015c710 legacy=0x7fb6d006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6b8003f90 0x7fb6b8004370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): picked mon.a con 0x7fb6b8003f90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6b8004a30 0x7fb6b8004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): picked mon.c con 0x7fb6b8004a30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 0x7fb6b800bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): picked mon.b con 0x7fb6b8005520 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b800c440 con 0x7fb6b8003f90 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b800c5f0 con 0x7fb6b8005520 2026-02-22T21:37:42.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b800c7f0 con 0x7fb6b8004a30 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d4f79640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d577a640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6b8004a30 0x7fb6b8004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d677c640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 0x7fb6b800bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d677c640 10 monclient(hunting): get_auth_request con 0x7fb6b8005520 auth_method 0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d677c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d677c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.400+0000 7fb6d677c640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): get_auth_request con 0x7fb6b8004a30 auth_method 0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d5f7b640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6b8003f90 0x7fb6b8004370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d5f7b640 10 monclient(hunting): get_auth_request con 0x7fb6b8003f90 auth_method 0 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d5f7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d5f7b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.401+0000 7fb6d5f7b640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 10 monclient(hunting): handle_auth_done global_id 4602 payload 293 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 -- 10.20.193.51:0/755240990 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6b8004a30 msgr2=0x7fb6b8004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6b8004a30 0x7fb6b8004eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 -- 10.20.193.51:0/755240990 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6b8003f90 msgr2=0x7fb6b8004370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6b8003f90 0x7fb6b8004370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 monclient: found mon.b 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 -- 10.20.193.51:0/755240990 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b8003620 con 0x7fb6b8005520 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d8206640 5 monclient: authenticate success, global_id 4602 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d8206640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.402+0000 7fb6d677c640 1 -- 10.20.193.51:0/755240990 reap_dead start 2026-02-22T21:37:42.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.403+0000 7fb6d677c640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 0x7fb6b800bb90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c2f0 tx=0x7fb6cc02f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f53a18319d4c55d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc0038d0 con 0x7fb6b8005520 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: dump: 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d8206640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6cc003c70 con 0x7fb6b8005520 2026-02-22T21:37:42.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 1 -- 10.20.193.51:0/755240990 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc005310 con 0x7fb6b8005520 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d4f79640 10 monclient: dump: 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d8206640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.404+0000 7fb6d8206640 4 set_mon_vals no callback set 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 10 monclient: shutdown 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 -- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 msgr2=0x7fb6b800bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 0x7fb6b800bb90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c2f0 tx=0x7fb6cc02f610 comp rx=0 tx=0).stop 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 -- 10.20.193.51:0/755240990 shutdown_connections 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 --2- 10.20.193.51:0/755240990 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6b8005520 0x7fb6b800bb90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 -- 10.20.193.51:0/755240990 >> 10.20.193.51:0/755240990 conn(0x7fb6d00cd7b0 msgr2=0x7fb6d00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 -- 10.20.193.51:0/755240990 shutdown_connections 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 1 -- 10.20.193.51:0/755240990 wait complete. 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.405+0000 7fb6d8206640 10 monclient: monmap: 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:42.420 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 librados: starting msgr at 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 librados: starting objecter 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 Processor -- start 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 -- start start 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 librados: setting wanted keys 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 librados: calling monclient init 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient: init 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 0x7fb6d01f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d014c800 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6d00bffd0 0x7fb6d01e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d00bffd0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d014cbe0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0150600 con 0x7fb6d014cbe0 2026-02-22T21:37:42.421 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d015bae0 con 0x7fb6d014c800 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0150780 con 0x7fb6d00bffd0 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d8206640 10 monclient(hunting): authenticate will time out at 1137.986665s 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.406+0000 7fb6d677c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6d00bffd0 0x7fb6d01e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33476/0 (socket says 10.20.193.51:33476) 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 1 -- 10.20.193.51:0/3926116685 learned_addr learned my addr 10.20.193.51:0/3926116685 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): get_auth_request con 0x7fb6d014cbe0 auth_method 0 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 0x7fb6d01f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): get_auth_request con 0x7fb6d00bffd0 auth_method 0 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): get_auth_request con 0x7fb6d014c800 auth_method 0 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d677c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d5f7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.407+0000 7fb6d577a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 -- 10.20.193.51:0/3926116685 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6d00bffd0 msgr2=0x7fb6d01e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6d00bffd0 0x7fb6d01e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 -- 10.20.193.51:0/3926116685 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 msgr2=0x7fb6d01f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 0x7fb6d01f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 monclient: found mon.noname-a 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d677c640 1 -- 10.20.193.51:0/3926116685 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d00bec50 con 0x7fb6d014cbe0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.408+0000 7fb6d5f7b640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 0x7fb6d01f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.409+0000 7fb6d677c640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c420 tx=0x7fb6cc002e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.409+0000 7fb6be7fc640 10 client.?.objecter ms_handle_connect 0x7fb6d014cbe0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.409+0000 7fb6be7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc046070 con 0x7fb6d014cbe0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: dump: 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.424 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6cc041640 con 0x7fb6d014cbe0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 5 monclient: authenticate success, global_id 4605 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 10 monclient: _renew_subs 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d00bef30 con 0x7fb6d014cbe0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc003220 con 0x7fb6d014cbe0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6be7fc640 10 monclient: dump: 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6bffff640 4 set_mon_vals no callback set 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 10 monclient: _renew_subs 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d00bf4f0 con 0x7fb6d014cbe0 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.411+0000 7fb6d8206640 1 librados: init done 2026-02-22T21:37:42.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.412+0000 7fb6d8206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.412+0000 7fb6d8206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.412+0000 7fb6d8206640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:42.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.412+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d00c73a0 con 0x7fb6d014cbe0 2026-02-22T21:37:42.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.423+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb6cc003ae0 con 0x7fb6d014cbe0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.424+0000 7fb6be7fc640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 0x7fb6a007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.424+0000 7fb6d5f7b640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 0x7fb6a007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.424+0000 7fb6d5f7b640 10 monclient: get_auth_request con 0x7fb6a0078e50 auth_method 0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.424+0000 7fb6be7fc640 10 client.4605.objecter ms_handle_connect 0x7fb6a0078e50 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.424+0000 7fb6d5f7b640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 0x7fb6a007b310 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0011570 tx=0x7fb6c0005050 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.425+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb6cc082770 con 0x7fb6d014cbe0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.425+0000 7fb6be7fc640 10 client.4605.objecter ms_dispatch2 0x7fb6d0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.425+0000 7fb6be7fc640 3 client.4605.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.425+0000 7fb6be7fc640 3 client.4605.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.425+0000 7fb6be7fc640 20 client.4605.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.528+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb6cc04f080 con 0x7fb6d014cbe0 2026-02-22T21:37:42.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.528+0000 7fb6be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.528+0000 7fb6be7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.612+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6d0153e70 con 0x7fb6a0078e50 2026-02-22T21:37:42.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.613+0000 7fb6be7fc640 1 -- 10.20.193.51:0/3926116685 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44546 (secure 0 0 0) 0x7fb6d0153e70 con 0x7fb6a0078e50 2026-02-22T21:37:42.615 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:42.615 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 10 librados: watch_flush enter 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 10 librados: watch_flush exit 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 20 client.4605.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 10 client.4605.objecter successfully canceled tick 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 msgr2=0x7fb6a007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 0x7fb6a007b310 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0011570 tx=0x7fb6c0005050 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 10 monclient: shutdown 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 msgr2=0x7fb6d0073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c420 tx=0x7fb6cc002e80 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 shutdown_connections 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb6a0078e50 0x7fb6a007b310 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6d014cbe0 0x7fb6d0073b30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb6d00bffd0 0x7fb6d01e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 --2- 10.20.193.51:0/3926116685 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb6d014c800 0x7fb6d01f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 >> 10.20.193.51:0/3926116685 conn(0x7fb6d00cd7b0 msgr2=0x7fb6d00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 shutdown_connections 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 -- 10.20.193.51:0/3926116685 wait complete. 2026-02-22T21:37:42.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.614+0000 7fb6d8206640 1 librados: shutdown 2026-02-22T21:37:42.624 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2026-02-22T21:37:41.723589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437164,"kb_used_data":5716,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509512276,"statfs":{"total":1546188226560,"available":1545740570624,"internally_reserved":0,"allocated":5853184,"data_stored":2836734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131107,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":18,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.864190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.821919+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:37:36.821919+0000","last_peered":"2026-02-22T21:37:36.821919+0000","last_clean":"2026-02-22T21:37:36.821919+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:37:36.821919+0000","last_undegraded":"2026-02-22T21:37:36.821919+0000","last_fullsized":"2026-02-22T21:37:36.821919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.774501+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:37:36.774501+0000","last_peered":"2026-02-22T21:37:36.774501+0000","last_clean":"2026-02-22T21:37:36.774501+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:37:36.774501+0000","last_undegraded":"2026-02-22T21:37:36.774501+0000","last_fullsized":"2026-02-22T21:37:36.774501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:38.356920+0000","last_change":"2026-02-22T21:37:38.380932+0000","last_active":"2026-02-22T21:37:38.356920+0000","last_peered":"2026-02-22T21:37:38.356920+0000","last_clean":"2026-02-22T21:37:38.356920+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:37:38.356920+0000","last_undegraded":"2026-02-22T21:37:38.356920+0000","last_fullsized":"2026-02-22T21:37:38.356920+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:22:09.624552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:37.091824+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:37:37.091824+0000","last_peered":"2026-02-22T21:37:37.091824+0000","last_clean":"2026-02-22T21:37:37.091824+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:37:37.091824+0000","last_undegraded":"2026-02-22T21:37:37.091824+0000","last_fullsized":"2026-02-22T21:37:37.091824+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.843543+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:37:36.843543+0000","last_peered":"2026-02-22T21:37:36.843543+0000","last_clean":"2026-02-22T21:37:36.843543+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:37:36.843543+0000","last_undegraded":"2026-02-22T21:37:36.843543+0000","last_fullsized":"2026-02-22T21:37:36.843543+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.800654+0000","last_change":"2026-02-22T21:37:36.829493+0000","last_active":"2026-02-22T21:37:36.800654+0000","last_peered":"2026-02-22T21:37:36.800654+0000","last_clean":"2026-02-22T21:37:36.800654+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:37:36.800654+0000","last_undegraded":"2026-02-22T21:37:36.800654+0000","last_fullsized":"2026-02-22T21:37:36.800654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:05.328006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:38.405885+0000","last_change":"2026-02-22T21:37:38.428511+0000","last_active":"2026-02-22T21:37:38.405885+0000","last_peered":"2026-02-22T21:37:38.405885+0000","last_clean":"2026-02-22T21:37:38.405885+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:37:38.405885+0000","last_undegraded":"2026-02-22T21:37:38.405885+0000","last_fullsized":"2026-02-22T21:37:38.405885+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:54:42.502365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.804957+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:37:36.804957+0000","last_peered":"2026-02-22T21:37:36.804957+0000","last_clean":"2026-02-22T21:37:36.804957+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:37:36.804957+0000","last_undegraded":"2026-02-22T21:37:36.804957+0000","last_fullsized":"2026-02-22T21:37:36.804957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:04:41.120271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"30'32","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:37.647400+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:37:37.647400+0000","last_peered":"2026-02-22T21:37:37.647400+0000","last_clean":"2026-02-22T21:37:37.647400+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:37:37.647400+0000","last_undegraded":"2026-02-22T21:37:37.647400+0000","last_fullsized":"2026-02-22T21:37:37.647400+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":327680,"data_stored":134320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":638976,"data_stored":452955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":327680,"data_stored":134320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27264,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":724992,"data_stored":532141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:37:42.625 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.691+0000 7fc34f5ce640 10 monclient: get_monmap_and_config 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.691+0000 7fc34f5ce640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.691+0000 7fc34f5ce640 10 monclient: monmap: 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.693 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:42.692332+0000 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:42.692332+0000 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:42.694 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 Processor -- start 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 -- start start 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient: init 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient: _add_conns ranks=[2,4,1,0,3,5] 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc348152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc34806d9d0 0x7fc3480c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-b con 0x7fc34806d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fc3480c93d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34806f1d0 con 0x7fc3480c93d0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34806de50 con 0x7fc348152ab0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc34806ed40 con 0x7fc34806d9d0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.692+0000 7fc34f5ce640 10 monclient(hunting): authenticate will time out at 1138.272735s 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34d343640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fc348152ab0 0x7fc3480c5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33372/0 (socket says 10.20.193.51:33372) 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34d343640 1 -- 10.20.193.51:0/46359909 learned_addr learned my addr 10.20.193.51:0/46359909 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34cb42640 1 --2- 10.20.193.51:0/46359909 >> v2:10.20.193.68:3300/0 conn(0x7fc34806d9d0 0x7fc3480c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34cb42640 10 monclient(hunting): get_auth_request con 0x7fc34806d9d0 auth_method 0 2026-02-22T21:37:42.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34cb42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34cb42640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.693+0000 7fc34cb42640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 1 -- 10.20.193.51:0/46359909 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110384779 0 0) 0x7fc34806de50 con 0x7fc348152ab0 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 10 monclient(hunting): my global_id is 4574 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.694+0000 7fc337fff640 1 -- 10.20.193.51:0/46359909 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc330003620 con 0x7fc348152ab0 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 10 monclient(hunting): handle_auth_done global_id 4609 payload 274 2026-02-22T21:37:42.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 1 -- 10.20.193.51:0/46359909 >> v1:10.20.193.125:6789/0 conn(0x7fc348152ab0 legacy=0x7fc3480c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 1 -- 10.20.193.51:0/46359909 >> v1:10.20.193.68:6789/0 conn(0x7fc3480c93d0 legacy=0x7fc348158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 1 monclient: found mon.noname-b 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.695+0000 7fc34cb42640 1 -- 10.20.193.51:0/46359909 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34806e130 con 0x7fc34806d9d0 2026-02-22T21:37:42.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.697+0000 7fc34cb42640 1 --2- 10.20.193.51:0/46359909 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3480c8e90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0098d0 tx=0x7fc33c02ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58f37e025007dc3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 1 -- 10.20.193.51:0/46359909 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c03c070 con 0x7fc34806d9d0 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: dump: 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.706 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 1 -- 10.20.193.51:0/46359909 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc33c02fbf0 con 0x7fc34806d9d0 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 1 -- 10.20.193.51:0/46359909 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c037440 con 0x7fc34806d9d0 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc337fff640 10 monclient: dump: 2026-02-22T21:37:42.707 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc34f5ce640 5 monclient: authenticate success, global_id 4609 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc34f5ce640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.698+0000 7fc34f5ce640 4 set_mon_vals no callback set 2026-02-22T21:37:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 10 monclient: shutdown 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 -- 10.20.193.51:0/46359909 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 msgr2=0x7fc3480c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/46359909 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3480c8e90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0098d0 tx=0x7fc33c02ef20 comp rx=0 tx=0).stop 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 -- 10.20.193.51:0/46359909 shutdown_connections 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/46359909 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3480c8e90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 -- 10.20.193.51:0/46359909 >> 10.20.193.51:0/46359909 conn(0x7fc3480cd7b0 msgr2=0x7fc3480cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 -- 10.20.193.51:0/46359909 shutdown_connections 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 -- 10.20.193.51:0/46359909 wait complete. 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 10 monclient: monmap: 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.709 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 librados: starting msgr at 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.699+0000 7fc34f5ce640 1 librados: starting objecter 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 Processor -- start 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 -- start start 2026-02-22T21:37:42.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 librados: setting wanted keys 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 librados: calling monclient init 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient: init 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-b con 0x7fc34806d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3480c93d0 0x7fc3481f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-c con 0x7fc3480c93d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc348152ab0 0x7fc3480c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): picked mon.noname-a con 0x7fc348152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc34806e6f0 con 0x7fc348152ab0 2026-02-22T21:37:42.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480c9880 con 0x7fc34806d9d0 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc34806e9d0 con 0x7fc3480c93d0 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34f5ce640 10 monclient(hunting): authenticate will time out at 1138.280462s 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34cb42640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3480c93d0 0x7fc3481f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34db44640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc348152ab0 0x7fc3480c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36164/0 (socket says 10.20.193.51:36164) 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 1 -- 10.20.193.51:0/951862438 learned_addr learned my addr 10.20.193.51:0/951862438 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 10 monclient(hunting): get_auth_request con 0x7fc34806d9d0 auth_method 0 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.700+0000 7fc34d343640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): get_auth_request con 0x7fc348152ab0 auth_method 0 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): get_auth_request con 0x7fc3480c93d0 auth_method 0 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:42.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34d343640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34d343640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34d343640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34db44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.701+0000 7fc34cb42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 10 monclient(hunting): handle_auth_done global_id 4615 payload 824 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 -- 10.20.193.51:0/951862438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3480c93d0 msgr2=0x7fc3481f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3480c93d0 0x7fc3481f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 -- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc348152ab0 msgr2=0x7fc3480c1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc348152ab0 0x7fc3480c1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 10 monclient: _finish_hunting 0 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 monclient: found mon.noname-b 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.702+0000 7fc34d343640 1 -- 10.20.193.51:0/951862438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3481593b0 con 0x7fc34806d9d0 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.703+0000 7fc34d343640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc33800c500 tx=0x7fc33800c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.703+0000 7fc335ffb640 10 client.?.objecter ms_handle_connect 0x7fc34806d9d0 2026-02-22T21:37:42.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.703+0000 7fc335ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338017070 con 0x7fc34806d9d0 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: dump: 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.717 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc338005de0 con 0x7fc34806d9d0 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:42.718 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3380072b0 con 0x7fc34806d9d0 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc335ffb640 10 monclient: dump: 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:42.719 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc3377fe640 4 set_mon_vals no callback set 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 5 monclient: authenticate success, global_id 4615 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 10 monclient: _renew_subs 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc348159690 con 0x7fc34806d9d0 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 10 monclient: _renew_subs 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3480c1390 con 0x7fc34806d9d0 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.705+0000 7fc34f5ce640 1 librados: init done 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.706+0000 7fc34f5ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.706+0000 7fc34f5ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.706+0000 7fc34f5ce640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:42.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.706+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc348152e90 con 0x7fc34806d9d0 2026-02-22T21:37:42.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc338007450 con 0x7fc34806d9d0 2026-02-22T21:37:42.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc335ffb640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 0x7fc31807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc34cb42640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 0x7fc31807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc34cb42640 10 monclient: get_auth_request con 0x7fc318078e50 auth_method 0 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc335ffb640 10 client.4615.objecter ms_handle_connect 0x7fc318078e50 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.717+0000 7fc34cb42640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 0x7fc31807b310 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc33c039000 tx=0x7fc33c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.718+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc338007770 con 0x7fc34806d9d0 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.718+0000 7fc335ffb640 10 client.4615.objecter ms_dispatch2 0x7fc348000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.718+0000 7fc335ffb640 3 client.4615.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.718+0000 7fc335ffb640 3 client.4615.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:42.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.718+0000 7fc335ffb640 20 client.4615.objecter dump_active .. 0 homeless 2026-02-22T21:37:42.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.819+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc338021020 con 0x7fc34806d9d0 2026-02-22T21:37:42.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.819+0000 7fc335ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:42.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.819+0000 7fc335ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:42.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.906+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3480c8480 con 0x7fc318078e50 2026-02-22T21:37:42.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.907+0000 7fc335ffb640 1 -- 10.20.193.51:0/951862438 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44546 (secure 0 0 0) 0x7fc3480c8480 con 0x7fc318078e50 2026-02-22T21:37:42.909 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:42.910 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:37:42.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 10 librados: watch_flush enter 2026-02-22T21:37:42.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 10 librados: watch_flush exit 2026-02-22T21:37:42.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 20 client.4615.objecter shutdown clearing up homeless session... 2026-02-22T21:37:42.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 10 client.4615.objecter successfully canceled tick 2026-02-22T21:37:42.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 msgr2=0x7fc31807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 0x7fc31807b310 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc33c039000 tx=0x7fc33c033000 comp rx=0 tx=0).stop 2026-02-22T21:37:42.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 10 monclient: shutdown 2026-02-22T21:37:42.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:42.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 msgr2=0x7fc3481567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc33800c500 tx=0x7fc33800c9d0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 shutdown_connections 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc318078e50 0x7fc31807b310 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc348152ab0 0x7fc3480c1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3480c93d0 0x7fc3481f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 --2- 10.20.193.51:0/951862438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc34806d9d0 0x7fc3481567d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 >> 10.20.193.51:0/951862438 conn(0x7fc3480cd7b0 msgr2=0x7fc3480c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 shutdown_connections 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 -- 10.20.193.51:0/951862438 wait complete. 2026-02-22T21:37:42.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.908+0000 7fc34f5ce640 1 librados: shutdown 2026-02-22T21:37:42.918 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2026-02-22T21:37:41.723589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437164,"kb_used_data":5716,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509512276,"statfs":{"total":1546188226560,"available":1545740570624,"internally_reserved":0,"allocated":5853184,"data_stored":2836734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131107,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":18,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.864190"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.821919+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:37:36.821919+0000","last_peered":"2026-02-22T21:37:36.821919+0000","last_clean":"2026-02-22T21:37:36.821919+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:37:36.821919+0000","last_undegraded":"2026-02-22T21:37:36.821919+0000","last_fullsized":"2026-02-22T21:37:36.821919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.774501+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:37:36.774501+0000","last_peered":"2026-02-22T21:37:36.774501+0000","last_clean":"2026-02-22T21:37:36.774501+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:37:36.774501+0000","last_undegraded":"2026-02-22T21:37:36.774501+0000","last_fullsized":"2026-02-22T21:37:36.774501+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:38.356920+0000","last_change":"2026-02-22T21:37:38.380932+0000","last_active":"2026-02-22T21:37:38.356920+0000","last_peered":"2026-02-22T21:37:38.356920+0000","last_clean":"2026-02-22T21:37:38.356920+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:37:38.356920+0000","last_undegraded":"2026-02-22T21:37:38.356920+0000","last_fullsized":"2026-02-22T21:37:38.356920+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:22:09.624552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:37.091824+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:37:37.091824+0000","last_peered":"2026-02-22T21:37:37.091824+0000","last_clean":"2026-02-22T21:37:37.091824+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:37:37.091824+0000","last_undegraded":"2026-02-22T21:37:37.091824+0000","last_fullsized":"2026-02-22T21:37:37.091824+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"30'2","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.843543+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:37:36.843543+0000","last_peered":"2026-02-22T21:37:36.843543+0000","last_clean":"2026-02-22T21:37:36.843543+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:37:36.843543+0000","last_undegraded":"2026-02-22T21:37:36.843543+0000","last_fullsized":"2026-02-22T21:37:36.843543+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.800654+0000","last_change":"2026-02-22T21:37:36.829493+0000","last_active":"2026-02-22T21:37:36.800654+0000","last_peered":"2026-02-22T21:37:36.800654+0000","last_clean":"2026-02-22T21:37:36.800654+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:37:36.800654+0000","last_undegraded":"2026-02-22T21:37:36.800654+0000","last_fullsized":"2026-02-22T21:37:36.800654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:05.328006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:38.405885+0000","last_change":"2026-02-22T21:37:38.428511+0000","last_active":"2026-02-22T21:37:38.405885+0000","last_peered":"2026-02-22T21:37:38.405885+0000","last_clean":"2026-02-22T21:37:38.405885+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:37:38.405885+0000","last_undegraded":"2026-02-22T21:37:38.405885+0000","last_fullsized":"2026-02-22T21:37:38.405885+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:54:42.502365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"27'1","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:36.804957+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:37:36.804957+0000","last_peered":"2026-02-22T21:37:36.804957+0000","last_clean":"2026-02-22T21:37:36.804957+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:37:36.804957+0000","last_undegraded":"2026-02-22T21:37:36.804957+0000","last_fullsized":"2026-02-22T21:37:36.804957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:04:41.120271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"30'32","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-02-22T21:37:37.647400+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:37:37.647400+0000","last_peered":"2026-02-22T21:37:37.647400+0000","last_clean":"2026-02-22T21:37:37.647400+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:37:37.647400+0000","last_undegraded":"2026-02-22T21:37:37.647400+0000","last_fullsized":"2026-02-22T21:37:37.647400+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215110,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215110,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":327680,"data_stored":134320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":638976,"data_stored":452955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":327680,"data_stored":134320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27264,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":724992,"data_stored":532141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":307200,"data_stored":124763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":323584,"data_stored":134301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:37:42.920 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T21:37:42.921 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-22T21:37:42.921 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-22T21:37:42.921 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:42.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.992+0000 7f50e3288640 10 monclient: get_monmap_and_config 2026-02-22T21:37:42.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.992+0000 7f50e3288640 10 monclient: build_initial_monmap 2026-02-22T21:37:42.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.992+0000 7f50e3288640 10 monclient: monmap: 2026-02-22T21:37:42.994 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:42.993308+0000 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:42.993308+0000 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:42.995 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:42.996 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 1 Processor -- start 2026-02-22T21:37:43.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 1 -- start start 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient: init 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient: _add_conns ranks=[2,0,1,5,4,3] 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50dc1526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50dc152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f50dc15d3b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50dc073ae0 con 0x7f50dc152a80 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50dc073d60 con 0x7f50dc15d3b0 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50dc074040 con 0x7f50dc1526a0 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:43.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.993+0000 7f50e3288640 10 monclient(hunting): authenticate will time out at 1138.573845s 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.994+0000 7f50d3fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f50dc152a80 0x7f50dc0cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47152/0 (socket says 10.20.193.51:47152) 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.994+0000 7f50d3fff640 1 -- 10.20.193.51:0/183675363 learned_addr learned my addr 10.20.193.51:0/183675363 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769311422 0 0) 0x7f50c4002920 con 0x7f50dc152a80 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _negotiate 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): my global_id is 4611 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c8003620 con 0x7f50dc152a80 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945659884 0 0) 0x7f50dc074040 con 0x7f50dc1526a0 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _negotiate 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): my global_id is 4580 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c4002920 con 0x7f50dc1526a0 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207511831 0 0) 0x7f50dc073d60 con 0x7f50dc15d3b0 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _negotiate 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 10 monclient(hunting): my global_id is 4618 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.995+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50dc074040 con 0x7f50dc15d3b0 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364510807 0 0) 0x7f50c4002920 con 0x7f50dc1526a0 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 >> v1:10.20.193.68:6789/0 conn(0x7f50dc15d3b0 legacy=0x7f50dc15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 >> v1:10.20.193.51:6789/0 conn(0x7f50dc152a80 legacy=0x7f50dc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:43.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc074320 con 0x7f50dc1526a0 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f50cc002ee0 con 0x7f50dc1526a0 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: dump: 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:43.006 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 >> v1:10.20.193.125:6789/0 conn(0x7f50dc1526a0 legacy=0x7f50dc0733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50c8003fb0 0x7f50c8004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): picked mon.c con 0x7f50c8003fb0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c80044c0 0x7f50c8004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): picked mon.a con 0x7f50c80044c0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 0x7f50c800bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): picked mon.b con 0x7f50c8009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f50c800c490 con 0x7f50c80044c0 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f50c800c610 con 0x7f50c8009ab0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f50c800c810 con 0x7f50c8003fb0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50d37fe640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.997+0000 7f50e17fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 0x7f50c800bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): get_auth_request con 0x7f50c8009ab0 auth_method 0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c80044c0 0x7f50c8004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): get_auth_request con 0x7f50c80044c0 auth_method 0 2026-02-22T21:37:43.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.998+0000 7f50e0ffd640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50c8003fb0 0x7f50c8004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): get_auth_request con 0x7f50c8003fb0 auth_method 0 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 10 monclient(hunting): handle_auth_done global_id 4580 payload 293 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 -- 10.20.193.51:0/183675363 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50c8003fb0 msgr2=0x7f50c8004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50c8003fb0 0x7f50c8004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 -- 10.20.193.51:0/183675363 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c80044c0 msgr2=0x7f50c8004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c80044c0 0x7f50c8004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 monclient: found mon.b 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 -- 10.20.193.51:0/183675363 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c4002920 con 0x7f50c8009ab0 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e3288640 5 monclient: authenticate success, global_id 4580 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e3288640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:42.999+0000 7f50e17fe640 1 -- 10.20.193.51:0/183675363 reap_dead start 2026-02-22T21:37:43.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.000+0000 7f50e17fe640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 0x7f50c800bea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f50d800c320 tx=0x7f50d802f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64c5716a2a467c6 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:43.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8003860 con 0x7f50c8009ab0 2026-02-22T21:37:43.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:43.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: dump: 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:43.019 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d8003c70 con 0x7f50c8009ab0 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 1 -- 10.20.193.51:0/183675363 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8005310 con 0x7f50c8009ab0 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 4 set_mon_vals no callback set 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50d37fe640 10 monclient: dump: 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.020 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 10 monclient: shutdown 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 -- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 msgr2=0x7f50c800bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 0x7f50c800bea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f50d800c320 tx=0x7f50d802f610 comp rx=0 tx=0).stop 2026-02-22T21:37:43.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 -- 10.20.193.51:0/183675363 shutdown_connections 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 --2- 10.20.193.51:0/183675363 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50c8009ab0 0x7f50c800bea0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 -- 10.20.193.51:0/183675363 >> 10.20.193.51:0/183675363 conn(0x7f50dc0ce350 msgr2=0x7f50dc0ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 -- 10.20.193.51:0/183675363 shutdown_connections 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 1 -- 10.20.193.51:0/183675363 wait complete. 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 10 monclient: build_initial_monmap 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.002+0000 7f50e3288640 10 monclient: monmap: 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:43.022 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 librados: starting msgr at 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 librados: starting objecter 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 Processor -- start 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 -- start start 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 librados: setting wanted keys 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 librados: calling monclient init 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient: init 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:43.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-a con 0x7f50c8009ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-b con 0x7f50dc1f3c60 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50dc1f40e0 0x7f50dc15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): picked mon.noname-c con 0x7f50dc1f40e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc074c50 con 0x7f50c8009ab0 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0cca00 con 0x7f50dc1f3c60 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:43.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0748e0 con 0x7f50dc1f40e0 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.003+0000 7f50e3288640 10 monclient(hunting): authenticate will time out at 1138.583804s 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50dc1f40e0 0x7f50dc15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33520/0 (socket says 10.20.193.51:33520) 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36186/0 (socket says 10.20.193.51:36186) 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 1 -- 10.20.193.51:0/2281381783 learned_addr learned my addr 10.20.193.51:0/2281381783 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): get_auth_request con 0x7f50c8009ab0 auth_method 0 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): get_auth_request con 0x7f50dc1f3c60 auth_method 0 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): get_auth_request con 0x7f50dc1f40e0 auth_method 0 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:43.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e0ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.004+0000 7f50e17fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 10 monclient(hunting): handle_auth_done global_id 4621 payload 824 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 -- 10.20.193.51:0/2281381783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50dc1f40e0 msgr2=0x7f50dc15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50dc1f40e0 0x7f50dc15ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 -- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 msgr2=0x7f50dc15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 monclient: found mon.noname-b 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50d3fff640 1 -- 10.20.193.51:0/2281381783 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc15b280 con 0x7f50dc1f3c60 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.005+0000 7f50e0ffd640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.007+0000 7f50d3fff640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007370 tx=0x7f50c4007840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.007+0000 7f50d17fa640 10 client.?.objecter ms_handle_connect 0x7f50dc1f3c60 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.007+0000 7f50d17fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c400f960 con 0x7f50dc1f3c60 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:43.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: dump: 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:43.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c4018070 con 0x7f50dc1f3c60 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c400fd40 con 0x7f50dc1f3c60 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d17fa640 10 monclient: dump: 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:43.030 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50d2ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50e3288640 5 monclient: authenticate success, global_id 4621 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50e3288640 10 monclient: _renew_subs 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50e3288640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.008+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc15b560 con 0x7f50dc1f3c60 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 monclient: _renew_subs 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc14c260 con 0x7f50dc1f3c60 2026-02-22T21:37:43.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 1 librados: init done 2026-02-22T21:37:43.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:43.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:43.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.009+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc15d820 con 0x7f50dc1f3c60 2026-02-22T21:37:43.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.112+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f50c401d460 con 0x7f50dc1f3c60 2026-02-22T21:37:43.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.112+0000 7f50d17fa640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 0x7f50ac07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:43.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.113+0000 7f50e0ffd640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 0x7f50ac07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:43.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.113+0000 7f50e0ffd640 10 monclient: get_auth_request con 0x7f50ac078e30 auth_method 0 2026-02-22T21:37:43.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.113+0000 7f50e0ffd640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 0x7f50ac07b2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0051e0 tx=0x7f50cc002ce0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:43.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.113+0000 7f50d17fa640 10 client.4621.objecter ms_handle_connect 0x7f50ac078e30 2026-02-22T21:37:43.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.114+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f50c405e340 con 0x7f50dc1f3c60 2026-02-22T21:37:43.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.114+0000 7f50d17fa640 10 client.4621.objecter ms_dispatch2 0x7f50dc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:43.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.114+0000 7f50d17fa640 3 client.4621.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:43.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.114+0000 7f50d17fa640 3 client.4621.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:43.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.114+0000 7f50d17fa640 20 client.4621.objecter dump_active .. 0 homeless 2026-02-22T21:37:43.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.120+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f50dc073ae0 con 0x7f50dc1f3c60 2026-02-22T21:37:43.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.120+0000 7f50d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:43.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.120+0000 7f50d17fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:43.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.220+0000 7f50e3288640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:43.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.220+0000 7f50e3288640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:43.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.220+0000 7f50e3288640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:43.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.220+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f50dc0c9fb0 con 0x7f50dc1f3c60 2026-02-22T21:37:43.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50d17fa640 1 -- 10.20.193.51:0/2281381783 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f50c4062300 con 0x7f50dc1f3c60 2026-02-22T21:37:43.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:43.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50d17fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:43.226 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:43.226 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:43.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 10 librados: watch_flush enter 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 10 librados: watch_flush exit 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 20 client.4621.objecter shutdown clearing up homeless session... 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 10 client.4621.objecter successfully canceled tick 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 msgr2=0x7f50ac07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 0x7f50ac07b2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0051e0 tx=0x7f50cc002ce0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 10 monclient: shutdown 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 msgr2=0x7f50dc15a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007370 tx=0x7f50c4007840 comp rx=0 tx=0).stop 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 shutdown_connections 2026-02-22T21:37:43.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f50ac078e30 0x7f50ac07b2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50dc1f40e0 0x7f50dc15ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50dc1f3c60 0x7f50dc15a740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 --2- 10.20.193.51:0/2281381783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f50c8009ab0 0x7f50dc15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 >> 10.20.193.51:0/2281381783 conn(0x7f50dc0ce350 msgr2=0x7f50dc151600 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 shutdown_connections 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 -- 10.20.193.51:0/2281381783 wait complete. 2026-02-22T21:37:43.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:43.224+0000 7f50e3288640 1 librados: shutdown 2026-02-22T21:37:46.236 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:46.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 10 monclient: get_monmap_and_config 2026-02-22T21:37:46.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 10 monclient: build_initial_monmap 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 10 monclient: monmap: 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:46.301747+0000 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:46.301747+0000 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:46.303 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:46.304 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:46.304 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 1 Processor -- start 2026-02-22T21:37:46.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 1 -- start start 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.301+0000 7f5f0d6bd640 10 monclient: init 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient: _add_conns ranks=[4,0,3,1,2,5] 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f08152ab0 0x7f5f080c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f5f08152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f0806d9d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5f080c95a0 0x7f5f08158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-a con 0x7f5f080c95a0 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0806f1d0 con 0x7f5f0806d9d0 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f5f0806ed40 con 0x7f5f080c95a0 2026-02-22T21:37:46.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f5f0806de50 con 0x7f5f08152ab0 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f0d6bd640 10 monclient(hunting): authenticate will time out at 1141.882171s 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f077fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5f080c95a0 0x7f5f08158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f08152ab0 0x7f5f080c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f08152ab0 0x7f5f080c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36188/0 (socket says 10.20.193.51:36188) 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 1 -- 10.20.193.51:0/39411852 learned_addr learned my addr 10.20.193.51:0/39411852 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 10 monclient(hunting): get_auth_request con 0x7f5f08152ab0 auth_method 0 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f06ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:46.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f077fe640 10 monclient(hunting): get_auth_request con 0x7f5f080c95a0 auth_method 0 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f077fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.302+0000 7f5f077fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.303+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_done global_id 4624 payload 274 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 1 -- 10.20.193.51:0/39411852 >> v2:10.20.193.51:3300/0 conn(0x7f5f080c95a0 msgr2=0x7f5f08158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:46.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 1 --2- 10.20.193.51:0/39411852 >> v2:10.20.193.51:3300/0 conn(0x7f5f080c95a0 0x7f5f08158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 1 -- 10.20.193.51:0/39411852 >> v1:10.20.193.51:6789/0 conn(0x7f5f0806d9d0 legacy=0x7f5f080c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 1 monclient: found mon.noname-b 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.304+0000 7f5f06ffd640 1 -- 10.20.193.51:0/39411852 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0806e130 con 0x7f5f08152ab0 2026-02-22T21:37:46.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.305+0000 7f5f06ffd640 1 --2- 10.20.193.51:0/39411852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f08152ab0 0x7f5f080c5ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009970 tx=0x7f5efc02efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a52405ef2f47ca server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 1 -- 10.20.193.51:0/39411852 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc03c070 con 0x7f5f08152ab0 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: dump: 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:46.315 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 1 -- 10.20.193.51:0/39411852 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5efc02fc80 con 0x7f5f08152ab0 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f0d6bd640 5 monclient: authenticate success, global_id 4624 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f0d6bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 1 -- 10.20.193.51:0/39411852 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc037660 con 0x7f5f08152ab0 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f0d6bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f0d6bd640 4 set_mon_vals no callback set 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.306+0000 7f5f05ffb640 10 monclient: dump: 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 10 monclient: shutdown 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/39411852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f08152ab0 msgr2=0x7f5f080c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/39411852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f08152ab0 0x7f5f080c5ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009970 tx=0x7f5efc02efb0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/39411852 shutdown_connections 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/39411852 >> v2:10.20.193.51:3300/0 conn(0x7f5f080c95a0 0x7f5f08158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/39411852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f08152ab0 0x7f5f080c5ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/39411852 >> 10.20.193.51:0/39411852 conn(0x7f5f080cd7b0 msgr2=0x7f5f080cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/39411852 shutdown_connections 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/39411852 wait complete. 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 10 monclient: build_initial_monmap 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 10 monclient: monmap: 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 librados: starting msgr at 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 librados: starting objecter 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 Processor -- start 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 -- start start 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 librados: setting wanted keys 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 1 librados: calling monclient init 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.307+0000 7f5f0d6bd640 10 monclient: init 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 0x7f5f081f5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0806d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f080c95a0 0x7f5f080c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-c con 0x7f5f080c95a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 0x7f5f08148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): picked mon.noname-a con 0x7f5f08152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0806e9d0 con 0x7f5f08152ab0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0806e6f0 con 0x7f5f0806d9d0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08152e90 con 0x7f5f080c95a0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f0d6bd640 10 monclient(hunting): authenticate will time out at 1141.888381s 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 0x7f5f08148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f06ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 0x7f5f081f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 0x7f5f08148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:33538/0 (socket says 10.20.193.51:33538) 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 1 -- 10.20.193.51:0/1326183515 learned_addr learned my addr 10.20.193.51:0/1326183515 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 10 monclient(hunting): get_auth_request con 0x7f5f08152ab0 auth_method 0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f077fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f06ffd640 10 monclient(hunting): get_auth_request con 0x7f5f0806d9d0 auth_method 0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f06ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.308+0000 7f5f06ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.309+0000 7f5f067fc640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f080c95a0 0x7f5f080c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f067fc640 10 monclient(hunting): get_auth_request con 0x7f5f080c95a0 auth_method 0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f067fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f067fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 10 monclient(hunting): handle_auth_done global_id 4627 payload 824 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 -- 10.20.193.51:0/1326183515 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f080c95a0 msgr2=0x7f5f080c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f080c95a0 0x7f5f080c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 -- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 msgr2=0x7f5f08148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 0x7f5f08148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 monclient: found mon.noname-b 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.310+0000 7f5f06ffd640 1 -- 10.20.193.51:0/1326183515 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f080c14e0 con 0x7f5f0806d9d0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.311+0000 7f5f06ffd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 0x7f5f081f5140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f0806eec0 tx=0x7f5efc0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.311+0000 7f5ee77fe640 10 client.?.objecter ms_handle_connect 0x7f5f0806d9d0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.311+0000 7f5ee77fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc03c040 con 0x7f5f0806d9d0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: dump: 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.318 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5efc004030 con 0x7f5f0806d9d0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc004420 con 0x7f5f0806d9d0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5ee77fe640 10 monclient: dump: 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 5 monclient: authenticate success, global_id 4627 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 10 monclient: _renew_subs 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f057fa640 4 set_mon_vals no callback set 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f080c17c0 con 0x7f5f0806d9d0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 10 monclient: _renew_subs 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f080c1d80 con 0x7f5f0806d9d0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.312+0000 7f5f0d6bd640 1 librados: init done 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.313+0000 7f5f0d6bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.313+0000 7f5f0d6bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.313+0000 7f5f0d6bd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.313+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0815d010 con 0x7f5f0806d9d0 2026-02-22T21:37:46.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f5efc003710 con 0x7f5f0806d9d0 2026-02-22T21:37:46.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5ee77fe640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 0x7f5ed407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:46.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5f067fc640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 0x7f5ed407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5f067fc640 10 monclient: get_auth_request con 0x7f5ed4078e00 auth_method 0 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5f067fc640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 0x7f5ed407b2c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f08150250 tx=0x7f5ef0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.324+0000 7f5ee77fe640 10 client.4627.objecter ms_handle_connect 0x7f5ed4078e00 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.325+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f5efc0045c0 con 0x7f5f0806d9d0 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.325+0000 7f5ee77fe640 10 client.4627.objecter ms_dispatch2 0x7f5f08000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:46.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.325+0000 7f5ee77fe640 3 client.4627.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:46.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.325+0000 7f5ee77fe640 3 client.4627.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:46.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.325+0000 7f5ee77fe640 20 client.4627.objecter dump_active .. 0 homeless 2026-02-22T21:37:46.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.424+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f5f0815d010 con 0x7f5f0806d9d0 2026-02-22T21:37:46.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.424+0000 7f5ee77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:46.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.424+0000 7f5ee77fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:46.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.524+0000 7f5f0d6bd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:46.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.524+0000 7f5f0d6bd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:46.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.524+0000 7f5f0d6bd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:37:46.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.524+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5f080c8e90 con 0x7f5f0806d9d0 2026-02-22T21:37:46.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.527+0000 7f5ee77fe640 1 -- 10.20.193.51:0/1326183515 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f5efc087c30 con 0x7f5f0806d9d0 2026-02-22T21:37:46.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.527+0000 7f5ee77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:46.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.527+0000 7f5ee77fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:46.529 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:46.529 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 10 librados: watch_flush enter 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 10 librados: watch_flush exit 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 20 client.4627.objecter shutdown clearing up homeless session... 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 10 client.4627.objecter successfully canceled tick 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 msgr2=0x7f5ed407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 0x7f5ed407b2c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5f08150250 tx=0x7f5ef0009040 comp rx=0 tx=0).stop 2026-02-22T21:37:46.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 10 monclient: shutdown 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 msgr2=0x7f5f081f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 0x7f5f081f5140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f0806eec0 tx=0x7f5efc0047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 shutdown_connections 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5ed4078e00 0x7f5ed407b2c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f08152ab0 0x7f5f08148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f080c95a0 0x7f5f080c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 --2- 10.20.193.51:0/1326183515 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f0806d9d0 0x7f5f081f5140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 >> 10.20.193.51:0/1326183515 conn(0x7f5f080cd7b0 msgr2=0x7f5f080cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 shutdown_connections 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 -- 10.20.193.51:0/1326183515 wait complete. 2026-02-22T21:37:46.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:46.528+0000 7f5f0d6bd640 1 librados: shutdown 2026-02-22T21:37:49.539 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:49.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.604+0000 7fa5f6d6a640 10 monclient: get_monmap_and_config 2026-02-22T21:37:49.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.604+0000 7fa5f6d6a640 10 monclient: build_initial_monmap 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.604+0000 7fa5f6d6a640 10 monclient: monmap: 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:49.605553+0000 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:49.605553+0000 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:49.607 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:49.608 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:49.608 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:49.608 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 1 Processor -- start 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 1 -- start start 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient: init 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient: _add_conns ranks=[5,0,3,2,1,4] 2026-02-22T21:37:49.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa5f00c9100 0x7fa5f00c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f00c9100 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5f006d9b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa5f00c6230 0x7fa5f0152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f00c6230 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f006e1c0 con 0x7fa5f006d9b0 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.605+0000 7fa5f6d6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f6d6a640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f006e4a0 con 0x7fa5f00c6230 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f6d6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f6d6a640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f006e780 con 0x7fa5f00c9100 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f6d6a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:49.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f6d6a640 10 monclient(hunting): authenticate will time out at 1145.186016s 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa5f00c9100 0x7fa5f00c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f52e0640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa5f00c6230 0x7fa5f0152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa5f00c9100 0x7fa5f00c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:41802/0 (socket says 10.20.193.51:41802) 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 1 -- 10.20.193.51:0/195411125 learned_addr learned my addr 10.20.193.51:0/195411125 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): get_auth_request con 0x7fa5f00c9100 auth_method 0 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:49.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f52e0640 10 monclient(hunting): get_auth_request con 0x7fa5f00c6230 auth_method 0 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f52e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f52e0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.606+0000 7fa5f52e0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_done global_id 4589 payload 274 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 1 -- 10.20.193.51:0/195411125 >> v2:10.20.193.51:3300/0 conn(0x7fa5f00c6230 msgr2=0x7fa5f0152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 1 --2- 10.20.193.51:0/195411125 >> v2:10.20.193.51:3300/0 conn(0x7fa5f00c6230 0x7fa5f0152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 1 -- 10.20.193.51:0/195411125 >> v1:10.20.193.51:6789/0 conn(0x7fa5f006d9b0 legacy=0x7fa5f00c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 10 monclient: _finish_hunting 0 2026-02-22T21:37:49.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 1 monclient: found mon.noname-c 2026-02-22T21:37:49.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:49.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:49.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.607+0000 7fa5f4adf640 1 -- 10.20.193.51:0/195411125 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f006ea60 con 0x7fa5f00c9100 2026-02-22T21:37:49.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.608+0000 7fa5f4adf640 1 --2- 10.20.193.51:0/195411125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f00c9100 0x7fa5f00c94e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0070da0 tx=0x7fa5e002ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc71391968f662f3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 1 -- 10.20.193.51:0/195411125 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e003c070 con 0x7fa5f00c9100 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: dump: 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:49.622 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 1 -- 10.20.193.51:0/195411125 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5e002fc70 con 0x7fa5f00c9100 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 5 monclient: authenticate success, global_id 4589 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:49.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 1 -- 10.20.193.51:0/195411125 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0037660 con 0x7fa5f00c9100 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 4 set_mon_vals no callback set 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5e77fe640 10 monclient: dump: 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:49.624 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 10 monclient: shutdown 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/195411125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f00c9100 msgr2=0x7fa5f00c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/195411125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f00c9100 0x7fa5f00c94e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0070da0 tx=0x7fa5e002ef60 comp rx=0 tx=0).stop 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/195411125 shutdown_connections 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/195411125 >> v2:10.20.193.51:3300/0 conn(0x7fa5f00c6230 0x7fa5f0152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/195411125 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f00c9100 0x7fa5f00c94e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/195411125 >> 10.20.193.51:0/195411125 conn(0x7fa5f00cd7b0 msgr2=0x7fa5f00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/195411125 shutdown_connections 2026-02-22T21:37:49.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.610+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/195411125 wait complete. 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 10 monclient: build_initial_monmap 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 10 monclient: monmap: 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:49.626 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 librados: starting msgr at 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 librados: starting objecter 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 Processor -- start 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 -- start start 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 librados: setting wanted keys 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 1 librados: calling monclient init 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.611+0000 7fa5f6d6a640 10 monclient: init 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f006d9b0 0x7fa5f0156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f006d9b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5f00c6230 0x7fa5f0157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f00c6230 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f00c9100 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f006ddb0 con 0x7fa5f00c9100 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f006f020 con 0x7fa5f00c6230 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f006df30 con 0x7fa5f006d9b0 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:49.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f6d6a640 10 monclient(hunting): authenticate will time out at 1145.192130s 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f006d9b0 0x7fa5f0156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5e7fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5f00c6230 0x7fa5f0157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:59960/0 (socket says 10.20.193.51:59960) 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 1 -- 10.20.193.51:0/1095357740 learned_addr learned my addr 10.20.193.51:0/1095357740 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): get_auth_request con 0x7fa5f00c9100 auth_method 0 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:49.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): get_auth_request con 0x7fa5f006d9b0 auth_method 0 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5e7fff640 10 monclient(hunting): get_auth_request con 0x7fa5f00c6230 auth_method 0 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5e7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5e7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:49.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.612+0000 7fa5f4adf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.613+0000 7fa5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.613+0000 7fa5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.613+0000 7fa5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 -- 10.20.193.51:0/1095357740 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f006d9b0 msgr2=0x7fa5f0156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f006d9b0 0x7fa5f0156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 -- 10.20.193.51:0/1095357740 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5f00c6230 msgr2=0x7fa5f0157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5f00c6230 0x7fa5f0157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 10 monclient: _finish_hunting 0 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 monclient: found mon.noname-a 2026-02-22T21:37:49.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.614+0000 7fa5f52e0640 1 -- 10.20.193.51:0/1095357740 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f0140e00 con 0x7fa5f00c9100 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.615+0000 7fa5f52e0640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00efc0 tx=0x7fa5ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.615+0000 7fa5e57fa640 10 client.?.objecter ms_handle_connect 0x7fa5f00c9100 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.615+0000 7fa5e57fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec018020 con 0x7fa5f00c9100 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: dump: 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.632 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 5 monclient: authenticate success, global_id 4632 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec014480 con 0x7fa5f00c9100 2026-02-22T21:37:49.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec014870 con 0x7fa5f00c9100 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _renew_subs 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f0157060 con 0x7fa5f00c9100 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e6ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5e57fa640 10 monclient: dump: 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:49.634 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _renew_subs 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f0159d40 con 0x7fa5f00c9100 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 1 librados: init done 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:49.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:49.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:49.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.617+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f00c94e0 con 0x7fa5f00c9100 2026-02-22T21:37:49.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.628+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa5ec014a10 con 0x7fa5f00c9100 2026-02-22T21:37:49.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.628+0000 7fa5e57fa640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 0x7fa5c007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.628+0000 7fa5f4adf640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 0x7fa5c007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.628+0000 7fa5f4adf640 10 monclient: get_auth_request con 0x7fa5c0078e80 auth_method 0 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.629+0000 7fa5f4adf640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 0x7fa5c007b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00c4430 tx=0x7fa5e0039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.629+0000 7fa5e57fa640 10 client.4632.objecter ms_handle_connect 0x7fa5c0078e80 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.630+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa5ec05ee80 con 0x7fa5f00c9100 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.630+0000 7fa5e57fa640 10 client.4632.objecter ms_dispatch2 0x7fa5f0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.630+0000 7fa5e57fa640 3 client.4632.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.630+0000 7fa5e57fa640 3 client.4632.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:49.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.630+0000 7fa5e57fa640 20 client.4632.objecter dump_active .. 0 homeless 2026-02-22T21:37:49.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.728+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa5ec022030 con 0x7fa5f00c9100 2026-02-22T21:37:49.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.728+0000 7fa5e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:49.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.728+0000 7fa5e57fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.829+0000 7fa5f6d6a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.829+0000 7fa5f6d6a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.829+0000 7fa5f6d6a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:49.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.829+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa5f00ca640 con 0x7fa5f00c9100 2026-02-22T21:37:49.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.835+0000 7fa5e57fa640 1 -- 10.20.193.51:0/1095357740 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa5ec010040 con 0x7fa5f00c9100 2026-02-22T21:37:49.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.835+0000 7fa5e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:49.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.835+0000 7fa5e57fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:49.837 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:49.837 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:49.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 10 librados: watch_flush enter 2026-02-22T21:37:49.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 10 librados: watch_flush exit 2026-02-22T21:37:49.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 20 client.4632.objecter shutdown clearing up homeless session... 2026-02-22T21:37:49.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 10 client.4632.objecter successfully canceled tick 2026-02-22T21:37:49.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 msgr2=0x7fa5c007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 0x7fa5c007b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00c4430 tx=0x7fa5e0039040 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 10 monclient: shutdown 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 msgr2=0x7fa5f0073860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00efc0 tx=0x7fa5ec00c490 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 shutdown_connections 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa5c0078e80 0x7fa5c007b340 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5f00c9100 0x7fa5f0073860 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5f00c6230 0x7fa5f0157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 --2- 10.20.193.51:0/1095357740 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5f006d9b0 0x7fa5f0156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:49.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 >> 10.20.193.51:0/1095357740 conn(0x7fa5f00cd7b0 msgr2=0x7fa5f0150e80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:49.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 shutdown_connections 2026-02-22T21:37:49.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 -- 10.20.193.51:0/1095357740 wait complete. 2026-02-22T21:37:49.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:49.836+0000 7fa5f6d6a640 1 librados: shutdown 2026-02-22T21:37:52.848 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 10 monclient: get_monmap_and_config 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 10 monclient: build_initial_monmap 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 10 monclient: monmap: 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:52.914774+0000 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:52.914774+0000 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:52.915 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 1 Processor -- start 2026-02-22T21:37:52.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 1 -- start start 2026-02-22T21:37:52.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.914+0000 7fae45abd640 10 monclient: init 2026-02-22T21:37:52.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:52.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient: _add_conns ranks=[5,0,1,2,3,4] 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fae40152ab0 0x7fae400c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-c con 0x7fae40152ab0 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae4006d9d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fae400c95a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae4006f1d0 con 0x7fae4006d9d0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae4006de50 con 0x7fae400c95a0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fae4006ed40 con 0x7fae40152ab0 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:52.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae45abd640 10 monclient(hunting): authenticate will time out at 1148.495116s 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3effd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fae4006d9d0 0x7fae400c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:55938/0 (socket says 10.20.193.51:55938) 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3effd640 1 -- 10.20.193.51:0/2823504322 learned_addr learned my addr 10.20.193.51:0/2823504322 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3f7fe640 1 --2- 10.20.193.51:0/2823504322 >> v2:10.20.193.125:3300/0 conn(0x7fae40152ab0 0x7fae400c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request con 0x7fae40152ab0 auth_method 0 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.915+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 1 -- 10.20.193.51:0/2823504322 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343459162 0 0) 0x7fae4006f1d0 con 0x7fae4006d9d0 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:37:52.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 10 monclient(hunting): my global_id is 4635 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.916+0000 7fae3e7fc640 1 -- 10.20.193.51:0/2823504322 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae24003620 con 0x7fae4006d9d0 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_done global_id 4595 payload 274 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 1 -- 10.20.193.51:0/2823504322 >> v1:10.20.193.68:6789/0 conn(0x7fae400c95a0 legacy=0x7fae40158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 1 -- 10.20.193.51:0/2823504322 >> v1:10.20.193.51:6789/0 conn(0x7fae4006d9d0 legacy=0x7fae400c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 1 monclient: found mon.noname-c 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:52.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.917+0000 7fae3f7fe640 1 -- 10.20.193.51:0/2823504322 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4006e130 con 0x7fae40152ab0 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.918+0000 7fae3f7fe640 1 --2- 10.20.193.51:0/2823504322 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae400c5ab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0098d0 tx=0x7fae2c02ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccee424028c94932 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 1 -- 10.20.193.51:0/2823504322 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c03c070 con 0x7fae40152ab0 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: dump: 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:52.922 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 1 -- 10.20.193.51:0/2823504322 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c02fbf0 con 0x7fae40152ab0 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 1 -- 10.20.193.51:0/2823504322 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c037440 con 0x7fae40152ab0 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae3e7fc640 10 monclient: dump: 2026-02-22T21:37:52.923 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae45abd640 5 monclient: authenticate success, global_id 4595 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae45abd640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.919+0000 7fae45abd640 4 set_mon_vals no callback set 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 10 monclient: shutdown 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 -- 10.20.193.51:0/2823504322 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 msgr2=0x7fae400c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 --2- 10.20.193.51:0/2823504322 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae400c5ab0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0098d0 tx=0x7fae2c02ef20 comp rx=0 tx=0).stop 2026-02-22T21:37:52.924 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 -- 10.20.193.51:0/2823504322 shutdown_connections 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 --2- 10.20.193.51:0/2823504322 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae400c5ab0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 -- 10.20.193.51:0/2823504322 >> 10.20.193.51:0/2823504322 conn(0x7fae400cd7b0 msgr2=0x7fae400cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 -- 10.20.193.51:0/2823504322 shutdown_connections 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 1 -- 10.20.193.51:0/2823504322 wait complete. 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 10 monclient: build_initial_monmap 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.920+0000 7fae45abd640 10 monclient: monmap: 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 librados: starting msgr at 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 librados: starting objecter 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 Processor -- start 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 -- start start 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 librados: setting wanted keys 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 librados: calling monclient init 2026-02-22T21:37:52.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient: init 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae4006d9d0 0x7fae401f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-b con 0x7fae4006d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-a con 0x7fae400c95a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae4015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:52.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): picked mon.noname-c con 0x7fae40152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fae4006e9d0 con 0x7fae400c95a0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fae4006e6f0 con 0x7fae4006d9d0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fae40152e90 con 0x7fae40152ab0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae45abd640 10 monclient(hunting): authenticate will time out at 1148.501637s 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:59972/0 (socket says 10.20.193.51:59972) 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 1 -- 10.20.193.51:0/1453981459 learned_addr learned my addr 10.20.193.51:0/1453981459 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 10 monclient(hunting): get_auth_request con 0x7fae400c95a0 auth_method 0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.921+0000 7fae3effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae4015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae4006d9d0 0x7fae401f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): get_auth_request con 0x7fae40152ab0 auth_method 0 2026-02-22T21:37:52.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request con 0x7fae4006d9d0 auth_method 0 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.922+0000 7fae3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 10 monclient(hunting): handle_auth_done global_id 4638 payload 824 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 -- 10.20.193.51:0/1453981459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 msgr2=0x7fae4015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae4015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 -- 10.20.193.51:0/1453981459 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae4006d9d0 msgr2=0x7fae401f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae4006d9d0 0x7fae401f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:52.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 10 monclient: _finish_hunting 0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 monclient: found mon.noname-a 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.923+0000 7fae3effd640 1 -- 10.20.193.51:0/1453981459 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae40148df0 con 0x7fae400c95a0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.924+0000 7fae3effd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae3000c520 tx=0x7fae3000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.924+0000 7fae1bfff640 10 client.?.objecter ms_handle_connect 0x7fae400c95a0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.924+0000 7fae1bfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30017070 con 0x7fae400c95a0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: dump: 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:52.932 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae30005de0 con 0x7fae400c95a0 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30007290 con 0x7fae400c95a0 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae1bfff640 10 monclient: dump: 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:52.933 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae3dffb640 4 set_mon_vals no callback set 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 5 monclient: authenticate success, global_id 4638 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _renew_subs 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae400c80c0 con 0x7fae400c95a0 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 client.4638.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _renew_subs 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae400c8680 con 0x7fae400c95a0 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 1 librados: init done 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:52.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:52.926+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae400c8e90 con 0x7fae400c95a0 2026-02-22T21:37:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.033+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fae30007430 con 0x7fae400c95a0 2026-02-22T21:37:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.034+0000 7fae1bfff640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 0x7fae1007b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.034+0000 7fae3f7fe640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 0x7fae1007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.034+0000 7fae3f7fe640 10 monclient: get_auth_request con 0x7fae10078e80 auth_method 0 2026-02-22T21:37:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.034+0000 7fae3f7fe640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 0x7fae1007b340 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae2c039000 tx=0x7fae2c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.034+0000 7fae1bfff640 10 client.4638.objecter ms_handle_connect 0x7fae10078e80 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fae30007750 con 0x7fae400c95a0 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 10 client.4638.objecter ms_dispatch2 0x7fae40000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 3 client.4638.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 3 client.4638.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 20 client.4638.objecter dump_active .. 0 homeless 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fae30021020 con 0x7fae400c95a0 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:53.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.041+0000 7fae1bfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.144+0000 7fae45abd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.144+0000 7fae45abd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.144+0000 7fae45abd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.144+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fae400c73b0 con 0x7fae400c95a0 2026-02-22T21:37:53.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.146+0000 7fae1bfff640 1 -- 10.20.193.51:0/1453981459 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fae30061410 con 0x7fae400c95a0 2026-02-22T21:37:53.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.146+0000 7fae1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:53.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.146+0000 7fae1bfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:53.148 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:53.148 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:53.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 10 librados: watch_flush enter 2026-02-22T21:37:53.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 10 librados: watch_flush exit 2026-02-22T21:37:53.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 20 client.4638.objecter shutdown clearing up homeless session... 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 10 client.4638.objecter successfully canceled tick 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 msgr2=0x7fae1007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 0x7fae1007b340 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae2c039000 tx=0x7fae2c033000 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 10 monclient: shutdown 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 msgr2=0x7fae401f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae3000c520 tx=0x7fae3000c9f0 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 shutdown_connections 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fae10078e80 0x7fae1007b340 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fae40152ab0 0x7fae4015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fae400c95a0 0x7fae401f4230 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 --2- 10.20.193.51:0/1453981459 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fae4006d9d0 0x7fae401f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 >> 10.20.193.51:0/1453981459 conn(0x7fae400cd7b0 msgr2=0x7fae400cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 shutdown_connections 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.147+0000 7fae45abd640 1 -- 10.20.193.51:0/1453981459 wait complete. 2026-02-22T21:37:53.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:53.148+0000 7fae45abd640 1 librados: shutdown 2026-02-22T21:37:56.158 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:56.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.228+0000 7ff7d355c640 10 monclient: get_monmap_and_config 2026-02-22T21:37:56.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.228+0000 7ff7d355c640 10 monclient: build_initial_monmap 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.228+0000 7ff7d355c640 10 monclient: monmap: 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:56.229629+0000 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:56.229629+0000 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:56.235 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 Processor -- start 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 -- start start 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient: init 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient: _add_conns ranks=[2,3,0,5,1,4] 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7cc150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff7cc15c710 0x7ff7cc0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-a con 0x7ff7cc15c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:37:56.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7cc150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7cc15a920 con 0x7ff7cc150600 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7cc15af60 con 0x7ff7cc150220 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff7cc15b240 con 0x7ff7cc15c710 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.229+0000 7ff7d355c640 10 monclient(hunting): authenticate will time out at 1151.809938s 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d12d1640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7ff7cc150220 0x7ff7cc15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:52944/0 (socket says 10.20.193.51:52944) 2026-02-22T21:37:56.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d12d1640 1 -- 10.20.193.51:0/2644145063 learned_addr learned my addr 10.20.193.51:0/2644145063 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d0ad0640 1 --2- 10.20.193.51:0/2644145063 >> v2:10.20.193.51:3300/0 conn(0x7ff7cc15c710 0x7ff7cc0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d0ad0640 10 monclient(hunting): get_auth_request con 0x7ff7cc15c710 auth_method 0 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d0ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d0ad0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.230+0000 7ff7d0ad0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:56.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 1 -- 10.20.193.51:0/2644145063 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960537943 0 0) 0x7ff7cc15af60 con 0x7ff7cc150220 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 10 monclient(hunting): _negotiate 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 10 monclient(hunting): my global_id is 4601 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.231+0000 7ff7bbfff640 1 -- 10.20.193.51:0/2644145063 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7b4003620 con 0x7ff7cc150220 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_done global_id 4641 payload 274 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 1 -- 10.20.193.51:0/2644145063 >> v1:10.20.193.125:6789/0 conn(0x7ff7cc150220 legacy=0x7ff7cc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 1 -- 10.20.193.51:0/2644145063 >> v1:10.20.193.51:6789/0 conn(0x7ff7cc150600 legacy=0x7ff7cc06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 10 monclient: _finish_hunting 0 2026-02-22T21:37:56.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 1 monclient: found mon.noname-a 2026-02-22T21:37:56.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:56.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.232+0000 7ff7d0ad0640 1 -- 10.20.193.51:0/2644145063 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc15b520 con 0x7ff7cc15c710 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.234+0000 7ff7d0ad0640 1 --2- 10.20.193.51:0/2644145063 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc15c710 0x7ff7cc0c73a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098d0 tx=0x7ff7c002ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd8d5b082087b54 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 1 -- 10.20.193.51:0/2644145063 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c003c070 con 0x7ff7cc15c710 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: dump: 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.250 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: _finish_auth 0 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: _check_auth_tickets 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 1 -- 10.20.193.51:0/2644145063 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c002fbf0 con 0x7ff7cc15c710 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 5 monclient: authenticate success, global_id 4641 2026-02-22T21:37:56.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 1 -- 10.20.193.51:0/2644145063 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037440 con 0x7ff7cc15c710 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 4 set_mon_vals no callback set 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7bbfff640 10 monclient: dump: 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.252 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 10 monclient: shutdown 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 -- 10.20.193.51:0/2644145063 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc15c710 msgr2=0x7ff7cc0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2644145063 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc15c710 0x7ff7cc0c73a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098d0 tx=0x7ff7c002ef20 comp rx=0 tx=0).stop 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 -- 10.20.193.51:0/2644145063 shutdown_connections 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2644145063 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc15c710 0x7ff7cc0c73a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 -- 10.20.193.51:0/2644145063 >> 10.20.193.51:0/2644145063 conn(0x7ff7cc0cd7b0 msgr2=0x7ff7cc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 -- 10.20.193.51:0/2644145063 shutdown_connections 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.236+0000 7ff7d355c640 1 -- 10.20.193.51:0/2644145063 wait complete. 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 10 monclient: build_initial_monmap 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 10 monclient: monmap: 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:56.254 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 librados: starting msgr at 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 librados: starting objecter 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 Processor -- start 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 -- start start 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 librados: setting wanted keys 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 1 librados: calling monclient init 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.237+0000 7ff7d355c640 10 monclient: init 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:56.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-a con 0x7ff7cc150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7cc150600 0x7ff7cc0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-c con 0x7ff7cc150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff7cc15c710 0x7ff7cc1f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): picked mon.noname-b con 0x7ff7cc15c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc0703b0 con 0x7ff7cc150220 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc15bae0 con 0x7ff7cc15c710 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc070530 con 0x7ff7cc150600 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d355c640 10 monclient(hunting): authenticate will time out at 1151.818336s 2026-02-22T21:37:56.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:59982/0 (socket says 10.20.193.51:59982) 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 1 -- 10.20.193.51:0/2585277759 learned_addr learned my addr 10.20.193.51:0/2585277759 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): get_auth_request con 0x7ff7cc150220 auth_method 0 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d0ad0640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7cc150600 0x7ff7cc0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d1ad2640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff7cc15c710 0x7ff7cc1f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:56.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d12d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d1ad2640 10 monclient(hunting): get_auth_request con 0x7ff7cc15c710 auth_method 0 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d1ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d1ad2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.238+0000 7ff7d1ad2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): get_auth_request con 0x7ff7cc150600 auth_method 0 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d1ad2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d1ad2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:56.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d1ad2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.239+0000 7ff7d0ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 10 monclient(hunting): handle_auth_done global_id 4647 payload 824 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 -- 10.20.193.51:0/2585277759 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7cc150600 msgr2=0x7ff7cc0c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7cc150600 0x7ff7cc0c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 -- 10.20.193.51:0/2585277759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff7cc15c710 msgr2=0x7ff7cc1f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff7cc15c710 0x7ff7cc1f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 10 monclient: _finish_hunting 0 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 monclient: found mon.noname-a 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:56.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.240+0000 7ff7d12d1640 1 -- 10.20.193.51:0/2585277759 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc0c2a90 con 0x7ff7cc150220 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.242+0000 7ff7d12d1640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c4d0 tx=0x7ff7bc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.242+0000 7ff7b9ffb640 10 client.?.objecter ms_handle_connect 0x7ff7cc150220 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.242+0000 7ff7b9ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc017070 con 0x7ff7cc150220 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: dump: 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.260 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: _finish_auth 0 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7bc005e00 con 0x7ff7cc150220 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 5 monclient: authenticate success, global_id 4647 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0072b0 con 0x7ff7cc150220 2026-02-22T21:37:56.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7bb7fe640 4 set_mon_vals no callback set 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7b9ffb640 10 monclient: dump: 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:56.262 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 monclient: _renew_subs 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7cc1f5cf0 con 0x7ff7cc150220 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 monclient: _renew_subs 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7cc0c2d00 con 0x7ff7cc150220 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 1 librados: init done 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.244+0000 7ff7d355c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.245+0000 7ff7d355c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.245+0000 7ff7d355c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.245+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7cc0c73a0 con 0x7ff7cc150220 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.256+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff7bc007450 con 0x7ff7cc150220 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.256+0000 7ff7b9ffb640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 0x7ff79c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.256+0000 7ff7d0ad0640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 0x7ff79c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.256+0000 7ff7d0ad0640 10 monclient: get_auth_request con 0x7ff79c078e50 auth_method 0 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.257+0000 7ff7d0ad0640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 0x7ff79c07b310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0039000 tx=0x7ff7c0033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.257+0000 7ff7b9ffb640 10 client.4647.objecter ms_handle_connect 0x7ff79c078e50 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.258+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff7bc007770 con 0x7ff7cc150220 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.258+0000 7ff7b9ffb640 10 client.4647.objecter ms_dispatch2 0x7ff7cc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.258+0000 7ff7b9ffb640 3 client.4647.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:56.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.258+0000 7ff7b9ffb640 3 client.4647.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:56.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.258+0000 7ff7b9ffb640 20 client.4647.objecter dump_active .. 0 homeless 2026-02-22T21:37:56.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.359+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff7bc021020 con 0x7ff7cc150220 2026-02-22T21:37:56.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.359+0000 7ff7b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:56.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.359+0000 7ff7b9ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:56.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.459+0000 7ff7d355c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:56.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.459+0000 7ff7d355c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:56.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.459+0000 7ff7d355c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:56.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.459+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff7cc14f450 con 0x7ff7cc150220 2026-02-22T21:37:56.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.461+0000 7ff7b9ffb640 1 -- 10.20.193.51:0/2585277759 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff7bc061420 con 0x7ff7cc150220 2026-02-22T21:37:56.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.461+0000 7ff7b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:56.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.461+0000 7ff7b9ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:56.463 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:56.463 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.462+0000 7ff7d355c640 10 librados: watch_flush enter 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 10 librados: watch_flush exit 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 20 client.4647.objecter shutdown clearing up homeless session... 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 10 client.4647.objecter successfully canceled tick 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 msgr2=0x7ff79c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 0x7ff79c07b310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0039000 tx=0x7ff7c0033000 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 10 monclient: shutdown 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 msgr2=0x7ff7cc06e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c4d0 tx=0x7ff7bc00c9a0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 shutdown_connections 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff79c078e50 0x7ff79c07b310 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff7cc15c710 0x7ff7cc1f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7cc150600 0x7ff7cc0c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 --2- 10.20.193.51:0/2585277759 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff7cc150220 0x7ff7cc06e8e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 >> 10.20.193.51:0/2585277759 conn(0x7ff7cc0cd7b0 msgr2=0x7ff7cc0c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 shutdown_connections 2026-02-22T21:37:56.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 -- 10.20.193.51:0/2585277759 wait complete. 2026-02-22T21:37:56.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:56.463+0000 7ff7d355c640 1 librados: shutdown 2026-02-22T21:37:59.474 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:37:59.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.539+0000 7f6f16f53640 10 monclient: get_monmap_and_config 2026-02-22T21:37:59.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.539+0000 7f6f16f53640 10 monclient: build_initial_monmap 2026-02-22T21:37:59.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.539+0000 7f6f16f53640 10 monclient: monmap: 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:37:59.540627+0000 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:37:59.540627+0000 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.547 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:37:59.548 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.540+0000 7f6f16f53640 1 Processor -- start 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.540+0000 7f6f16f53640 1 -- start start 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.540+0000 7f6f16f53640 10 monclient: init 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.540+0000 7f6f16f53640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient: _add_conns ranks=[5,0,2,3,4,1] 2026-02-22T21:37:59.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6f10150220 0x7f6f100c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-c con 0x7f6f10150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f10150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f1015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1015a920 con 0x7f6f10150600 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1015af60 con 0x7f6f1015c710 2026-02-22T21:37:59.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1015b240 con 0x7f6f10150220 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f16f53640 10 monclient(hunting): authenticate will time out at 1155.121063s 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f07fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f6f10150600 0x7f6f1015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:53044/0 (socket says 10.20.193.51:53044) 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f07fff640 1 -- 10.20.193.51:0/2635647622 learned_addr learned my addr 10.20.193.51:0/2635647622 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f14cc8640 1 --2- 10.20.193.51:0/2635647622 >> v2:10.20.193.125:3300/0 conn(0x7f6f10150220 0x7f6f100c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:59.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f14cc8640 10 monclient(hunting): get_auth_request con 0x7f6f10150220 auth_method 0 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f14cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f14cc8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.541+0000 7f6f14cc8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:59.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 1 -- 10.20.193.51:0/2635647622 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4118429763 0 0) 0x7f6f1015a920 con 0x7f6f10150600 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 10 monclient(hunting): _negotiate 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 10 monclient(hunting): my global_id is 4650 2026-02-22T21:37:59.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.542+0000 7f6f077fe640 1 -- 10.20.193.51:0/2635647622 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6efc003620 con 0x7f6f10150600 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2635647622 >> v1:10.20.193.125:6789/0 conn(0x7f6f1015c710 legacy=0x7f6f1006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2635647622 >> v1:10.20.193.51:6789/0 conn(0x7f6f10150600 legacy=0x7f6f1015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 10 monclient: _finish_hunting 0 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 1 monclient: found mon.noname-c 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:59.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:37:59.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.543+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2635647622 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1015b520 con 0x7f6f10150220 2026-02-22T21:37:59.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.544+0000 7f6f14cc8640 1 --2- 10.20.193.51:0/2635647622 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f10150220 0x7f6f100c73a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f000098a0 tx=0x7f6f0002eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0874fcde4f68fe0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:59.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 1 -- 10.20.193.51:0/2635647622 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0003c070 con 0x7f6f10150220 2026-02-22T21:37:59.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:59.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: dump: 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.569 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: _finish_auth 0 2026-02-22T21:37:59.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: _check_auth_tickets 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 5 monclient: authenticate success, global_id 4607 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 1 -- 10.20.193.51:0/2635647622 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f0002fb70 con 0x7f6f10150220 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 1 -- 10.20.193.51:0/2635647622 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00037440 con 0x7f6f10150220 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:59.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f077fe640 10 monclient: dump: 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.572 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 10 monclient: get_monmap_and_config success 2026-02-22T21:37:59.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 4 set_mon_vals no callback set 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 10 monclient: shutdown 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 -- 10.20.193.51:0/2635647622 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f10150220 msgr2=0x7f6f100c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2635647622 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f10150220 0x7f6f100c73a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6f000098a0 tx=0x7f6f0002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 -- 10.20.193.51:0/2635647622 shutdown_connections 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2635647622 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f10150220 0x7f6f100c73a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 -- 10.20.193.51:0/2635647622 >> 10.20.193.51:0/2635647622 conn(0x7f6f100cd7b0 msgr2=0x7f6f100cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 -- 10.20.193.51:0/2635647622 shutdown_connections 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.546+0000 7f6f16f53640 1 -- 10.20.193.51:0/2635647622 wait complete. 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 10 monclient: build_initial_monmap 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 10 monclient: monmap: 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:37:59.575 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 librados: starting msgr at 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 librados: starting objecter 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 Processor -- start 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 -- start start 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 librados: setting wanted keys 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 1 librados: calling monclient init 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.547+0000 7f6f16f53640 10 monclient: init 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient: _reopen_session rank -1 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:37:59.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-a con 0x7f6f10150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f10150600 0x7f6f1006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-b con 0x7f6f10150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1015c710 0x7f6f101542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): picked mon.noname-c con 0x7f6f1015c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6f100703b0 con 0x7f6f10150220 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1015bae0 con 0x7f6f10150600 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10070530 con 0x7f6f1015c710 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): _renew_subs 2026-02-22T21:37:59.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f16f53640 10 monclient(hunting): authenticate will time out at 1155.128327s 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42938/0 (socket says 10.20.193.51:42938) 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2892804745 learned_addr learned my addr 10.20.193.51:0/2892804745 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): get_auth_request con 0x7f6f10150220 auth_method 0 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f154c9640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1015c710 0x7f6f101542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f154c9640 10 monclient(hunting): get_auth_request con 0x7f6f1015c710 auth_method 0 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f154c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f154c9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f07fff640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f10150600 0x7f6f1006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:59.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f154c9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f07fff640 10 monclient(hunting): get_auth_request con 0x7f6f10150600 auth_method 0 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.548+0000 7f6f07fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f07fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f154c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f154c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f154c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:37:59.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.549+0000 7f6f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 10 monclient(hunting): handle_auth_done global_id 4653 payload 824 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2892804745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1015c710 msgr2=0x7f6f101542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1015c710 0x7f6f101542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2892804745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f10150600 msgr2=0x7f6f1006ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f10150600 0x7f6f1006ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 10 monclient: _finish_hunting 0 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 monclient: found mon.noname-a 2026-02-22T21:37:59.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.550+0000 7f6f14cc8640 1 -- 10.20.193.51:0/2892804745 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1006cf70 con 0x7f6f10150220 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.551+0000 7f6f14cc8640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f000099d0 tx=0x7f6f000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.551+0000 7f6f057fa640 10 client.?.objecter ms_handle_connect 0x7f6f10150220 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.551+0000 7f6f057fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00046020 con 0x7f6f10150220 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: dump: 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:59.581 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: _finish_auth 0 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: _check_auth_tickets 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00042620 con 0x7f6f10150220 2026-02-22T21:37:59.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0003c040 con 0x7f6f10150220 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f06ffd640 4 set_mon_vals no callback set 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f057fa640 10 monclient: dump: 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 5 monclient: authenticate success, global_id 4653 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _renew_subs 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1006d230 con 0x7f6f10150220 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _renew_subs 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1006d7f0 con 0x7f6f10150220 2026-02-22T21:37:59.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 1 librados: init done 2026-02-22T21:37:59.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:59.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:59.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:59.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.553+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f100c73a0 con 0x7f6f10150220 2026-02-22T21:37:59.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.658+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6f0003f030 con 0x7f6f10150220 2026-02-22T21:37:59.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.659+0000 7f6f057fa640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 0x7f6ee007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:37:59.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.659+0000 7f6f07fff640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 0x7f6ee007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:37:59.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.659+0000 7f6f07fff640 10 monclient: get_auth_request con 0x7f6ee0078e00 auth_method 0 2026-02-22T21:37:59.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.659+0000 7f6f057fa640 10 client.4653.objecter ms_handle_connect 0x7f6ee0078e00 2026-02-22T21:37:59.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.659+0000 7f6f07fff640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 0x7f6ee007b2c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8007b60 tx=0x7f6ef8007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:37:59.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.666+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f6f0002fa30 con 0x7f6f10150220 2026-02-22T21:37:59.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.666+0000 7f6f057fa640 10 client.4653.objecter ms_dispatch2 0x7f6f10000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:37:59.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.666+0000 7f6f057fa640 3 client.4653.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:37:59.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.666+0000 7f6f057fa640 3 client.4653.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:37:59.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.667+0000 7f6f057fa640 20 client.4653.objecter dump_active .. 0 homeless 2026-02-22T21:37:59.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.667+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6f00050070 con 0x7f6f10150220 2026-02-22T21:37:59.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.667+0000 7f6f057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:37:59.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.667+0000 7f6f057fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:37:59.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.765+0000 7f6f16f53640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:37:59.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.765+0000 7f6f16f53640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:59.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.765+0000 7f6f16f53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:37:59.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.765+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6f10074da0 con 0x7f6f10150220 2026-02-22T21:37:59.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.768+0000 7f6f057fa640 1 -- 10.20.193.51:0/2892804745 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f6f000871a0 con 0x7f6f10150220 2026-02-22T21:37:59.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.768+0000 7f6f057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:37:59.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.768+0000 7f6f057fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:37:59.771 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:37:59.771 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:37:59.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 10 librados: watch_flush enter 2026-02-22T21:37:59.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 10 librados: watch_flush exit 2026-02-22T21:37:59.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 20 client.4653.objecter shutdown clearing up homeless session... 2026-02-22T21:37:59.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 10 client.4653.objecter successfully canceled tick 2026-02-22T21:37:59.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 msgr2=0x7f6ee007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 0x7f6ee007b2c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8007b60 tx=0x7f6ef8007ab0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 10 monclient: shutdown 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 msgr2=0x7f6f101f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f000099d0 tx=0x7f6f000047c0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 shutdown_connections 2026-02-22T21:37:59.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6ee0078e00 0x7f6ee007b2c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6f1015c710 0x7f6f101542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6f10150600 0x7f6f1006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 --2- 10.20.193.51:0/2892804745 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6f10150220 0x7f6f101f2100 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 >> 10.20.193.51:0/2892804745 conn(0x7f6f100cd7b0 msgr2=0x7f6f1015c000 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 shutdown_connections 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.769+0000 7f6f16f53640 1 -- 10.20.193.51:0/2892804745 wait complete. 2026-02-22T21:37:59.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:37:59.770+0000 7f6f16f53640 1 librados: shutdown 2026-02-22T21:38:01.248 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:32.402 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.250 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:32.401 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.250 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:32.404 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.251 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:32.403 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.263 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:32.417 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.264 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:32.415 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.277 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:32.392 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.280 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:32.396 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:01.290 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:32.406 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:02.781 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:02.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 10 monclient: get_monmap_and_config 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 10 monclient: build_initial_monmap 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 10 monclient: monmap: 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:02.846933+0000 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:02.846933+0000 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:02.852 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 1 Processor -- start 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 1 -- start start 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.846+0000 7fa8e4e69640 10 monclient: init 2026-02-22T21:38:02.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient: _add_conns ranks=[5,3,4,0,1,2] 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa8e0150220 0x7fa8e00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e0150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa8e0150600 0x7fa8e015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e0150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa8e015c540 0x7fa8e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e015c540 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:02.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e015c9b0 con 0x7fa8e0150600 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e015a920 con 0x7fa8e015c540 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e015af60 con 0x7fa8e0150220 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:02.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8e4e69640 10 monclient(hunting): authenticate will time out at 1158.427245s 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa8e0150600 0x7fa8e015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa8e0150600 0x7fa8e015c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42940/0 (socket says 10.20.193.51:42940) 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 1 -- 10.20.193.51:0/1664074886 learned_addr learned my addr 10.20.193.51:0/1664074886 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): get_auth_request con 0x7fa8e0150600 auth_method 0 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ded76640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.68:3300/0 conn(0x7fa8e015c540 0x7fa8e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8de575640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.125:3300/0 conn(0x7fa8e0150220 0x7fa8e00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ded76640 10 monclient(hunting): get_auth_request con 0x7fa8e015c540 auth_method 0 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ded76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ded76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ded76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.847+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): get_auth_request con 0x7fa8e0150220 auth_method 0 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_done global_id 4656 payload 274 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 -- 10.20.193.51:0/1664074886 >> v2:10.20.193.125:3300/0 conn(0x7fa8e0150220 msgr2=0x7fa8e00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:02.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.125:3300/0 conn(0x7fa8e0150220 0x7fa8e00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 -- 10.20.193.51:0/1664074886 >> v2:10.20.193.68:3300/0 conn(0x7fa8e015c540 msgr2=0x7fa8e006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.68:3300/0 conn(0x7fa8e015c540 0x7fa8e006fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 10 monclient: _finish_hunting 0 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 monclient: found mon.noname-a 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.848+0000 7fa8ddd74640 1 -- 10.20.193.51:0/1664074886 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e015b240 con 0x7fa8e0150600 2026-02-22T21:38:02.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.849+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/1664074886 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150600 0x7fa8e015c000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00c65c0 tx=0x7fa8c802ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef08f84bafaa2fa6 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:02.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 1 -- 10.20.193.51:0/1664074886 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c803c070 con 0x7fa8e0150600 2026-02-22T21:38:02.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:02.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:02.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: dump: 2026-02-22T21:38:02.868 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: _finish_auth 0 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: _check_auth_tickets 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 1 -- 10.20.193.51:0/1664074886 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c802fbd0 con 0x7fa8e0150600 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 1 -- 10.20.193.51:0/1664074886 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8037440 con 0x7fa8e0150600 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8dd573640 10 monclient: dump: 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.870 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8e4e69640 5 monclient: authenticate success, global_id 4656 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8e4e69640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.851+0000 7fa8e4e69640 4 set_mon_vals no callback set 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 10 monclient: shutdown 2026-02-22T21:38:02.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 -- 10.20.193.51:0/1664074886 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150600 msgr2=0x7fa8e015c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/1664074886 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150600 0x7fa8e015c000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00c65c0 tx=0x7fa8c802ef50 comp rx=0 tx=0).stop 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 -- 10.20.193.51:0/1664074886 shutdown_connections 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.68:3300/0 conn(0x7fa8e015c540 0x7fa8e006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/1664074886 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150600 0x7fa8e015c000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/1664074886 >> v2:10.20.193.125:3300/0 conn(0x7fa8e0150220 0x7fa8e00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 -- 10.20.193.51:0/1664074886 >> 10.20.193.51:0/1664074886 conn(0x7fa8e00cd7b0 msgr2=0x7fa8e00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 -- 10.20.193.51:0/1664074886 shutdown_connections 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 1 -- 10.20.193.51:0/1664074886 wait complete. 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 10 monclient: build_initial_monmap 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.852+0000 7fa8e4e69640 10 monclient: monmap: 2026-02-22T21:38:02.872 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 librados: starting msgr at 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 librados: starting objecter 2026-02-22T21:38:02.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 Processor -- start 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 -- start start 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 librados: setting wanted keys 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 librados: calling monclient init 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient: init 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e0150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa8e0150600 0x7fa8e006e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e0150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa8e015c540 0x7fa8e01575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e015c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e015bae0 con 0x7fa8e0150220 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e015b800 con 0x7fa8e015c540 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e00701e0 con 0x7fa8e0150600 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8e4e69640 10 monclient(hunting): authenticate will time out at 1158.433543s 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8ded76640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa8e015c540 0x7fa8e01575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42948/0 (socket says 10.20.193.51:42948) 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 1 -- 10.20.193.51:0/125871222 learned_addr learned my addr 10.20.193.51:0/125871222 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 10 monclient(hunting): get_auth_request con 0x7fa8e0150220 auth_method 0 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8de575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.853+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa8e0150600 0x7fa8e006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): get_auth_request con 0x7fa8e015c540 auth_method 0 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8de575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): get_auth_request con 0x7fa8e0150600 auth_method 0 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ded76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:02.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.854+0000 7fa8ddd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 10 monclient(hunting): handle_auth_done global_id 4659 payload 824 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 -- 10.20.193.51:0/125871222 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa8e0150600 msgr2=0x7fa8e006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa8e0150600 0x7fa8e006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 -- 10.20.193.51:0/125871222 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa8e015c540 msgr2=0x7fa8e01575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa8e015c540 0x7fa8e01575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 10 monclient: _finish_hunting 0 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 monclient: found mon.noname-a 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:02.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.855+0000 7fa8de575640 1 -- 10.20.193.51:0/125871222 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e01f7790 con 0x7fa8e0150220 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.856+0000 7fa8de575640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400ec10 tx=0x7fa8d400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.856+0000 7fa8c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa8e0150220 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.856+0000 7fa8c6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4017070 con 0x7fa8e0150220 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: dump: 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:02.879 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d400cc40 con 0x7fa8e0150220 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4013470 con 0x7fa8e0150220 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8c6ffd640 10 monclient: dump: 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:02.881 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8dcd72640 4 set_mon_vals no callback set 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 5 monclient: authenticate success, global_id 4659 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _renew_subs 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e01f7a70 con 0x7fa8e0150220 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:02.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _renew_subs 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e01f3a90 con 0x7fa8e0150220 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 1 librados: init done 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:02.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.858+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e015e110 con 0x7fa8e0150220 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.871+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa8d4013630 con 0x7fa8e0150220 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.871+0000 7fa8c6ffd640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 0x7fa8ac07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.872+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 0x7fa8ac07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.872+0000 7fa8ddd74640 10 monclient: get_auth_request con 0x7fa8ac078e00 auth_method 0 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.872+0000 7fa8c6ffd640 10 client.4659.objecter ms_handle_connect 0x7fa8ac078e00 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.872+0000 7fa8ddd74640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 0x7fa8ac07b2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0074940 tx=0x7fa8c8039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.873+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa8d405e520 con 0x7fa8e0150220 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.873+0000 7fa8c6ffd640 10 client.4659.objecter ms_dispatch2 0x7fa8e0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.873+0000 7fa8c6ffd640 3 client.4659.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:02.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.873+0000 7fa8c6ffd640 3 client.4659.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:02.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.873+0000 7fa8c6ffd640 20 client.4659.objecter dump_active .. 0 homeless 2026-02-22T21:38:02.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.974+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa8d4097e20 con 0x7fa8e0150220 2026-02-22T21:38:02.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.974+0000 7fa8c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:02.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:02.974+0000 7fa8c6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:03.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.073+0000 7fa8e4e69640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:03.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.073+0000 7fa8e4e69640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:03.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.073+0000 7fa8e4e69640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:03.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.073+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa8e0070570 con 0x7fa8e0150220 2026-02-22T21:38:03.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.075+0000 7fa8c6ffd640 1 -- 10.20.193.51:0/125871222 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa8e015e110 con 0x7fa8e0150220 2026-02-22T21:38:03.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.075+0000 7fa8c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:03.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.075+0000 7fa8c6ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:03.077 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:03.077 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 10 librados: watch_flush enter 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 10 librados: watch_flush exit 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 20 client.4659.objecter shutdown clearing up homeless session... 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 10 client.4659.objecter successfully canceled tick 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 msgr2=0x7fa8ac07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 0x7fa8ac07b2c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0074940 tx=0x7fa8c8039040 comp rx=0 tx=0).stop 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 10 monclient: shutdown 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:03.079 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 msgr2=0x7fa8e01f4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400ec10 tx=0x7fa8d400c520 comp rx=0 tx=0).stop 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 shutdown_connections 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa8ac078e00 0x7fa8ac07b2c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa8e015c540 0x7fa8e01575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa8e0150600 0x7fa8e006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 --2- 10.20.193.51:0/125871222 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa8e0150220 0x7fa8e01f4570 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 >> 10.20.193.51:0/125871222 conn(0x7fa8e00cd7b0 msgr2=0x7fa8e006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 shutdown_connections 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 -- 10.20.193.51:0/125871222 wait complete. 2026-02-22T21:38:03.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:03.077+0000 7fa8e4e69640 1 librados: shutdown 2026-02-22T21:38:06.087 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:06.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 10 monclient: get_monmap_and_config 2026-02-22T21:38:06.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 10 monclient: build_initial_monmap 2026-02-22T21:38:06.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 10 monclient: monmap: 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:06.156035+0000 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:06.156035+0000 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:06.157 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 1 Processor -- start 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 1 -- start start 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.155+0000 7fde27596640 10 monclient: init 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient: _add_conns ranks=[0,3,1,5,2,4] 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde20150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fde2015c710 0x7fde200c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): picked mon.noname-a con 0x7fde2015c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fde20150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2015a920 con 0x7fde20150220 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2015af60 con 0x7fde20150600 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fde2015b240 con 0x7fde2015c710 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde27596640 10 monclient(hunting): authenticate will time out at 1161.736462s 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.156+0000 7fde24b0a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fde2015c710 0x7fde200c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde25b0c640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fde20150600 0x7fde2006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:33174/0 (socket says 10.20.193.51:33174) 2026-02-22T21:38:06.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde25b0c640 1 -- 10.20.193.51:0/3089886351 learned_addr learned my addr 10.20.193.51:0/3089886351 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): get_auth_request con 0x7fde2015c710 auth_method 0 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.157+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 1 -- 10.20.193.51:0/3089886351 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342886106 0 0) 0x7fde2015af60 con 0x7fde20150600 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 10 monclient(hunting): _negotiate 2026-02-22T21:38:06.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 10 monclient(hunting): my global_id is 4651 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde0ffff640 1 -- 10.20.193.51:0/3089886351 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde08003620 con 0x7fde20150600 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 10 monclient(hunting): handle_auth_done global_id 4662 payload 274 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 1 -- 10.20.193.51:0/3089886351 >> v1:10.20.193.68:6789/0 conn(0x7fde20150600 legacy=0x7fde2006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 1 -- 10.20.193.51:0/3089886351 >> v1:10.20.193.51:6789/0 conn(0x7fde20150220 legacy=0x7fde2015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 10 monclient: _finish_hunting 0 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 1 monclient: found mon.noname-a 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.158+0000 7fde24b0a640 1 -- 10.20.193.51:0/3089886351 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2015b520 con 0x7fde2015c710 2026-02-22T21:38:06.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.160+0000 7fde24b0a640 1 --2- 10.20.193.51:0/3089886351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde2015c710 0x7fde200c73a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fde140098d0 tx=0x7fde1402ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3fda20ed164aeb server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 1 -- 10.20.193.51:0/3089886351 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1403c070 con 0x7fde2015c710 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: dump: 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:06.164 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: _finish_auth 0 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 1 -- 10.20.193.51:0/3089886351 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1402fbf0 con 0x7fde2015c710 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 5 monclient: authenticate success, global_id 4662 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 1 -- 10.20.193.51:0/3089886351 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14037440 con 0x7fde2015c710 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde0ffff640 10 monclient: dump: 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:06.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 4 set_mon_vals no callback set 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 10 monclient: shutdown 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 -- 10.20.193.51:0/3089886351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde2015c710 msgr2=0x7fde200c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 --2- 10.20.193.51:0/3089886351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde2015c710 0x7fde200c73a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fde140098d0 tx=0x7fde1402ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 -- 10.20.193.51:0/3089886351 shutdown_connections 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 --2- 10.20.193.51:0/3089886351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde2015c710 0x7fde200c73a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 -- 10.20.193.51:0/3089886351 >> 10.20.193.51:0/3089886351 conn(0x7fde200cd7b0 msgr2=0x7fde200cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 -- 10.20.193.51:0/3089886351 shutdown_connections 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.163+0000 7fde27596640 1 -- 10.20.193.51:0/3089886351 wait complete. 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient: build_initial_monmap 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient: monmap: 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 librados: starting msgr at 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 librados: starting objecter 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 Processor -- start 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 -- start start 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 librados: setting wanted keys 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 librados: calling monclient init 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient: init 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fde20150220 0x7fde2014b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): picked mon.noname-b con 0x7fde20150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): picked mon.noname-a con 0x7fde20150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:06.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fde2015c710 0x7fde200bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): picked mon.noname-c con 0x7fde2015c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fde200703b0 con 0x7fde20150600 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fde2015bae0 con 0x7fde20150220 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fde20070530 con 0x7fde2015c710 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.164+0000 7fde27596640 10 monclient(hunting): authenticate will time out at 1161.744946s 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42958/0 (socket says 10.20.193.51:42958) 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 1 -- 10.20.193.51:0/1130810264 learned_addr learned my addr 10.20.193.51:0/1130810264 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): get_auth_request con 0x7fde20150600 auth_method 0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fde20150220 0x7fde2014b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde24b0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde25b0c640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fde2015c710 0x7fde200bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): get_auth_request con 0x7fde20150220 auth_method 0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde25b0c640 10 monclient(hunting): get_auth_request con 0x7fde2015c710 auth_method 0 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde25b0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde25b0c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:06.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde25b0c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.165+0000 7fde2530b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde25b0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde25b0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:06.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde25b0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 -- 10.20.193.51:0/1130810264 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fde2015c710 msgr2=0x7fde200bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fde2015c710 0x7fde200bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 -- 10.20.193.51:0/1130810264 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fde20150220 msgr2=0x7fde2014b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fde20150220 0x7fde2014b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 10 monclient: _finish_hunting 0 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 monclient: found mon.noname-a 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.166+0000 7fde24b0a640 1 -- 10.20.193.51:0/1130810264 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde200be9b0 con 0x7fde20150600 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.167+0000 7fde24b0a640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde1403a040 tx=0x7fde140047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.167+0000 7fde0dffb640 10 client.?.objecter ms_handle_connect 0x7fde20150600 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.167+0000 7fde0dffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14046020 con 0x7fde20150600 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: dump: 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.171 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: _finish_auth 0 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde14042650 con 0x7fde20150600 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1403c040 con 0x7fde20150600 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 5 monclient: authenticate success, global_id 4668 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _renew_subs 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2014cf70 con 0x7fde20150600 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0f7fe640 4 set_mon_vals no callback set 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde0dffb640 10 monclient: dump: 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:06.172 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _renew_subs 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde200bec50 con 0x7fde20150600 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 1 librados: init done 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.169+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde20074da0 con 0x7fde20150600 2026-02-22T21:38:06.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.180+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fde1404a3b0 con 0x7fde20150600 2026-02-22T21:38:06.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.181+0000 7fde0dffb640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 0x7fddf007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:06.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.181+0000 7fde2530b640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 0x7fddf007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:06.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.181+0000 7fde2530b640 10 monclient: get_auth_request con 0x7fddf0078e00 auth_method 0 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.181+0000 7fde2530b640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 0x7fddf007b2c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde10002770 tx=0x7fde10007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.181+0000 7fde0dffb640 10 client.4668.objecter ms_handle_connect 0x7fddf0078e00 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.182+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fde1402fb30 con 0x7fde20150600 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.182+0000 7fde0dffb640 10 client.4668.objecter ms_dispatch2 0x7fde20000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.182+0000 7fde0dffb640 3 client.4668.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.182+0000 7fde0dffb640 3 client.4668.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:06.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.182+0000 7fde0dffb640 20 client.4668.objecter dump_active .. 0 homeless 2026-02-22T21:38:06.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.284+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fde14050020 con 0x7fde20150600 2026-02-22T21:38:06.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.284+0000 7fde0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:06.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.284+0000 7fde0dffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:06.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.384+0000 7fde27596640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:06.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.384+0000 7fde27596640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:06.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.384+0000 7fde27596640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:06.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.384+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fde2014ef60 con 0x7fde20150600 2026-02-22T21:38:06.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.387+0000 7fde0dffb640 1 -- 10.20.193.51:0/1130810264 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fde14086a20 con 0x7fde20150600 2026-02-22T21:38:06.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.387+0000 7fde0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:06.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.387+0000 7fde0dffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:06.389 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:06.389 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:06.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 10 librados: watch_flush enter 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 10 librados: watch_flush exit 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 20 client.4668.objecter shutdown clearing up homeless session... 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 10 client.4668.objecter successfully canceled tick 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 msgr2=0x7fddf007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 0x7fddf007b2c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde10002770 tx=0x7fde10007ef0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 10 monclient: shutdown 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 msgr2=0x7fde200bf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde1403a040 tx=0x7fde140047c0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 shutdown_connections 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fddf0078e00 0x7fddf007b2c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fde2015c710 0x7fde200bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fde20150600 0x7fde200bf730 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 --2- 10.20.193.51:0/1130810264 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fde20150220 0x7fde2014b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 >> 10.20.193.51:0/1130810264 conn(0x7fde200cd7b0 msgr2=0x7fde200c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 shutdown_connections 2026-02-22T21:38:06.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 -- 10.20.193.51:0/1130810264 wait complete. 2026-02-22T21:38:06.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:06.389+0000 7fde27596640 1 librados: shutdown 2026-02-22T21:38:09.399 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:09.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 10 monclient: get_monmap_and_config 2026-02-22T21:38:09.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 10 monclient: build_initial_monmap 2026-02-22T21:38:09.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 10 monclient: monmap: 2026-02-22T21:38:09.467 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:09.467 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:09.465932+0000 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:09.465932+0000 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:09.468 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 1 Processor -- start 2026-02-22T21:38:09.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 1 -- start start 2026-02-22T21:38:09.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.464+0000 7f35b7f11640 10 monclient: init 2026-02-22T21:38:09.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:09.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient: _add_conns ranks=[3,4,1,0,5,2] 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f35b0150220 0x7f35b00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-a con 0x7f35b0150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f35b0150600 0x7f35b015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-b con 0x7f35b0150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f35b015c540 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b015c9b0 con 0x7f35b015c540 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f35b015a920 con 0x7f35b0150220 2026-02-22T21:38:09.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f35b015af60 con 0x7f35b0150600 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b7f11640 10 monclient(hunting): authenticate will time out at 1165.046258s 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f35b0150220 0x7f35b00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5485640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f35b0150600 0x7f35b015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f35b0150220 0x7f35b00c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50874/0 (socket says 10.20.193.51:50874) 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 1 -- 10.20.193.51:0/69182562 learned_addr learned my addr 10.20.193.51:0/69182562 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 10 monclient(hunting): get_auth_request con 0x7f35b0150220 auth_method 0 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:09.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5c86640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5485640 10 monclient(hunting): get_auth_request con 0x7f35b0150600 auth_method 0 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5485640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5485640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.465+0000 7f35b5485640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:09.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.466+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 10 monclient(hunting): handle_auth_done global_id 4671 payload 274 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 1 -- 10.20.193.51:0/69182562 >> v2:10.20.193.68:3300/0 conn(0x7f35b0150600 msgr2=0x7f35b015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 1 --2- 10.20.193.51:0/69182562 >> v2:10.20.193.68:3300/0 conn(0x7f35b0150600 0x7f35b015c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 1 -- 10.20.193.51:0/69182562 >> v1:10.20.193.68:6789/0 conn(0x7f35b015c540 legacy=0x7f35b006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 10 monclient: _finish_hunting 0 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 1 monclient: found mon.noname-a 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.467+0000 7f35b5c86640 1 -- 10.20.193.51:0/69182562 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b015b240 con 0x7f35b0150220 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.468+0000 7f35b5c86640 1 --2- 10.20.193.51:0/69182562 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150220 0x7f35b00c73a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f35b0074940 tx=0x7f35a002ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a304c4b8909627 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:09.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 1 -- 10.20.193.51:0/69182562 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a003c070 con 0x7f35b0150220 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: dump: 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:09.482 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: _finish_auth 0 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: _check_auth_tickets 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 5 monclient: authenticate success, global_id 4671 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 1 -- 10.20.193.51:0/69182562 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a002fbd0 con 0x7f35b0150220 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 1 -- 10.20.193.51:0/69182562 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0037440 con 0x7f35b0150220 2026-02-22T21:38:09.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b4c84640 10 monclient: dump: 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 4 set_mon_vals no callback set 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 10 monclient: shutdown 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 -- 10.20.193.51:0/69182562 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150220 msgr2=0x7f35b00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 --2- 10.20.193.51:0/69182562 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150220 0x7f35b00c73a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f35b0074940 tx=0x7f35a002ef60 comp rx=0 tx=0).stop 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 -- 10.20.193.51:0/69182562 shutdown_connections 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 --2- 10.20.193.51:0/69182562 >> v2:10.20.193.68:3300/0 conn(0x7f35b0150600 0x7f35b015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 --2- 10.20.193.51:0/69182562 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150220 0x7f35b00c73a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 -- 10.20.193.51:0/69182562 >> 10.20.193.51:0/69182562 conn(0x7f35b00cd7b0 msgr2=0x7f35b00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 -- 10.20.193.51:0/69182562 shutdown_connections 2026-02-22T21:38:09.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 1 -- 10.20.193.51:0/69182562 wait complete. 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 10 monclient: build_initial_monmap 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.470+0000 7f35b7f11640 10 monclient: monmap: 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:09.486 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 librados: starting msgr at 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 librados: starting objecter 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 Processor -- start 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 -- start start 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 librados: setting wanted keys 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 librados: calling monclient init 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient: init 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:09.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f35b0150220 0x7f35b014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-c con 0x7f35b0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-a con 0x7f35b0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f35b015c540 0x7f35b01f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): picked mon.noname-b con 0x7f35b015c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:09.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f35b015bae0 con 0x7f35b0150600 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f35b015b800 con 0x7f35b015c540 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f35b00703b0 con 0x7f35b0150220 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.471+0000 7f35b7f11640 10 monclient(hunting): authenticate will time out at 1165.052867s 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f35b015c540 0x7f35b01f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f35b0150220 0x7f35b014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50876/0 (socket says 10.20.193.51:50876) 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 1 -- 10.20.193.51:0/1013211471 learned_addr learned my addr 10.20.193.51:0/1013211471 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): get_auth_request con 0x7f35b0150600 auth_method 0 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:09.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): get_auth_request con 0x7f35b015c540 auth_method 0 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5485640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): get_auth_request con 0x7f35b0150220 auth_method 0 2026-02-22T21:38:09.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b6487640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:09.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.472+0000 7f35b5c86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 10 monclient(hunting): handle_auth_done global_id 4674 payload 824 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 -- 10.20.193.51:0/1013211471 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f35b0150220 msgr2=0x7f35b014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f35b0150220 0x7f35b014cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 -- 10.20.193.51:0/1013211471 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f35b015c540 msgr2=0x7f35b01f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f35b015c540 0x7f35b01f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 10 monclient: _finish_hunting 0 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 monclient: found mon.noname-a 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:09.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.473+0000 7f35b5485640 1 -- 10.20.193.51:0/1013211471 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b0159b60 con 0x7f35b0150600 2026-02-22T21:38:09.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.475+0000 7f35b5485640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35a400ebe0 tx=0x7f35a400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:09.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.475+0000 7f359e7fc640 10 client.?.objecter ms_handle_connect 0x7f35b0150600 2026-02-22T21:38:09.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.475+0000 7f359e7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:09.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4017070 con 0x7f35b0150600 2026-02-22T21:38:09.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: dump: 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.499 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:09.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a400cc40 con 0x7f35b0150600 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4013470 con 0x7f35b0150600 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359e7fc640 10 monclient: dump: 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:09.501 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f359ffff640 4 set_mon_vals no callback set 2026-02-22T21:38:09.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 5 monclient: authenticate success, global_id 4674 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 10 monclient: _renew_subs 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b01587e0 con 0x7f35b0150600 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 10 monclient: _renew_subs 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b0158da0 con 0x7f35b0150600 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.476+0000 7f35b7f11640 1 librados: init done 2026-02-22T21:38:09.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.477+0000 7f35b7f11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.477+0000 7f35b7f11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.477+0000 7f35b7f11640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.477+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b00ccad0 con 0x7f35b0150600 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.489+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f35a4024060 con 0x7f35b0150600 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.490+0000 7f359e7fc640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 0x7f358007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.490+0000 7f35b5c86640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 0x7f358007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.490+0000 7f35b5c86640 10 monclient: get_auth_request con 0x7f3580078e00 auth_method 0 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.490+0000 7f359e7fc640 10 client.4674.objecter ms_handle_connect 0x7f3580078e00 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.490+0000 7f35b5c86640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 0x7f358007b2c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f35b00c65c0 tx=0x7f35a0033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:09.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.491+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f35a405de80 con 0x7f35b0150600 2026-02-22T21:38:09.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.491+0000 7f359e7fc640 10 client.4674.objecter ms_dispatch2 0x7f35b0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:09.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.491+0000 7f359e7fc640 3 client.4674.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:09.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.491+0000 7f359e7fc640 3 client.4674.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:09.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.491+0000 7f359e7fc640 20 client.4674.objecter dump_active .. 0 homeless 2026-02-22T21:38:09.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.593+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f35b00ccad0 con 0x7f35b0150600 2026-02-22T21:38:09.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.593+0000 7f359e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:09.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.593+0000 7f359e7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:09.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.692+0000 7f35b7f11640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:09.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.692+0000 7f35b7f11640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:09.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.692+0000 7f35b7f11640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:09.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.692+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f35b00c0be0 con 0x7f35b0150600 2026-02-22T21:38:09.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.694+0000 7f359e7fc640 1 -- 10.20.193.51:0/1013211471 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f35b00c0be0 con 0x7f35b0150600 2026-02-22T21:38:09.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.694+0000 7f359e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:09.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.694+0000 7f359e7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:09.697 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:09.697 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:09.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.695+0000 7f35b7f11640 10 librados: watch_flush enter 2026-02-22T21:38:09.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.695+0000 7f35b7f11640 10 librados: watch_flush exit 2026-02-22T21:38:09.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.695+0000 7f35b7f11640 20 client.4674.objecter shutdown clearing up homeless session... 2026-02-22T21:38:09.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.695+0000 7f35b7f11640 10 client.4674.objecter successfully canceled tick 2026-02-22T21:38:09.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.695+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 msgr2=0x7f358007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 0x7f358007b2c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f35b00c65c0 tx=0x7f35a0033000 comp rx=0 tx=0).stop 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 10 monclient: shutdown 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 msgr2=0x7f35b0154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35a400ebe0 tx=0x7f35a400c520 comp rx=0 tx=0).stop 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 shutdown_connections 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3580078e00 0x7f358007b2c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f35b015c540 0x7f35b01f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f35b0150600 0x7f35b0154220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 --2- 10.20.193.51:0/1013211471 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f35b0150220 0x7f35b014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:09.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 >> 10.20.193.51:0/1013211471 conn(0x7f35b00cd7b0 msgr2=0x7f35b00cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:09.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 shutdown_connections 2026-02-22T21:38:09.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 -- 10.20.193.51:0/1013211471 wait complete. 2026-02-22T21:38:09.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:09.696+0000 7f35b7f11640 1 librados: shutdown 2026-02-22T21:38:12.707 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.773+0000 7fe043103640 10 monclient: get_monmap_and_config 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.773+0000 7fe043103640 10 monclient: build_initial_monmap 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.773+0000 7fe043103640 10 monclient: monmap: 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:12.775320+0000 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:12.775320+0000 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 Processor -- start 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 -- start start 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient: init 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient: _add_conns ranks=[3,2,1,5,4,0] 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe03c152ab0 0x7fe03c0c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:12.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): picked mon.noname-a con 0x7fe03c152ab0 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe03c06d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe03c0c95a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03c06f1d0 con 0x7fe03c0c95a0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03c06de50 con 0x7fe03c06d9d0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fe03c06ed40 con 0x7fe03c152ab0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.774+0000 7fe043103640 10 monclient(hunting): authenticate will time out at 1168.355661s 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe03c152ab0 0x7fe03c0c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe041679640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fe03c0c95a0 0x7fe03c158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:43604/0 (socket says 10.20.193.51:43604) 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe041679640 1 -- 10.20.193.51:0/2872230637 learned_addr learned my addr 10.20.193.51:0/2872230637 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): get_auth_request con 0x7fe03c152ab0 auth_method 0 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:12.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.775+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 1 -- 10.20.193.51:0/2872230637 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131462846 0 0) 0x7fe03c06f1d0 con 0x7fe03c0c95a0 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 10 monclient(hunting): _negotiate 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 10 monclient(hunting): my global_id is 4663 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe0337fe640 1 -- 10.20.193.51:0/2872230637 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe028003620 con 0x7fe03c0c95a0 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 10 monclient(hunting): handle_auth_done global_id 4677 payload 274 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 1 -- 10.20.193.51:0/2872230637 >> v1:10.20.193.125:6789/0 conn(0x7fe03c06d9d0 legacy=0x7fe03c0c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:12.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 1 -- 10.20.193.51:0/2872230637 >> v1:10.20.193.68:6789/0 conn(0x7fe03c0c95a0 legacy=0x7fe03c158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 10 monclient: _finish_hunting 0 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 1 monclient: found mon.noname-a 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.776+0000 7fe040e78640 1 -- 10.20.193.51:0/2872230637 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c06e130 con 0x7fe03c152ab0 2026-02-22T21:38:12.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.777+0000 7fe040e78640 1 --2- 10.20.193.51:0/2872230637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c152ab0 0x7fe03c0c5ab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0098d0 tx=0x7fe02c02ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a524529152c2c77f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 1 -- 10.20.193.51:0/2872230637 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c03c070 con 0x7fe03c152ab0 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: dump: 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:12.781 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 1 -- 10.20.193.51:0/2872230637 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe02c02fbf0 con 0x7fe03c152ab0 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 5 monclient: authenticate success, global_id 4677 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 1 -- 10.20.193.51:0/2872230637 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c037440 con 0x7fe03c152ab0 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 4 set_mon_vals no callback set 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe0337fe640 10 monclient: dump: 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 10 monclient: shutdown 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 -- 10.20.193.51:0/2872230637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c152ab0 msgr2=0x7fe03c0c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 --2- 10.20.193.51:0/2872230637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c152ab0 0x7fe03c0c5ab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe02c0098d0 tx=0x7fe02c02ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 -- 10.20.193.51:0/2872230637 shutdown_connections 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 --2- 10.20.193.51:0/2872230637 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c152ab0 0x7fe03c0c5ab0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 -- 10.20.193.51:0/2872230637 >> 10.20.193.51:0/2872230637 conn(0x7fe03c0cd7b0 msgr2=0x7fe03c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 -- 10.20.193.51:0/2872230637 shutdown_connections 2026-02-22T21:38:12.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.779+0000 7fe043103640 1 -- 10.20.193.51:0/2872230637 wait complete. 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient: build_initial_monmap 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient: monmap: 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:12.783 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 librados: starting msgr at 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 librados: starting objecter 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 Processor -- start 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 -- start start 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 librados: setting wanted keys 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 librados: calling monclient init 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient: init 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe03c06d9d0 0x7fe03c15b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): picked mon.noname-c con 0x7fe03c06d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): picked mon.noname-a con 0x7fe03c0c95a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe03c152ab0 0x7fe03c1efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): picked mon.noname-b con 0x7fe03c152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c06e9d0 con 0x7fe03c0c95a0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c06e6f0 con 0x7fe03c152ab0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c152e90 con 0x7fe03c06d9d0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.780+0000 7fe043103640 10 monclient(hunting): authenticate will time out at 1168.361826s 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe03c152ab0 0x7fe03c1efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50902/0 (socket says 10.20.193.51:50902) 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 1 -- 10.20.193.51:0/3435100402 learned_addr learned my addr 10.20.193.51:0/3435100402 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): get_auth_request con 0x7fe03c0c95a0 auth_method 0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe03c06d9d0 0x7fe03c15b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): get_auth_request con 0x7fe03c152ab0 auth_method 0 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:12.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe033fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): get_auth_request con 0x7fe03c06d9d0 auth_method 0 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe041679640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:12.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.781+0000 7fe040e78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 -- 10.20.193.51:0/3435100402 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe03c06d9d0 msgr2=0x7fe03c15b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe03c06d9d0 0x7fe03c15b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 -- 10.20.193.51:0/3435100402 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe03c152ab0 msgr2=0x7fe03c1efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe03c152ab0 0x7fe03c1efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 10 monclient: _finish_hunting 0 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 monclient: found mon.noname-a 2026-02-22T21:38:12.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:12.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:12.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.782+0000 7fe033fff640 1 -- 10.20.193.51:0/3435100402 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c1f2b70 con 0x7fe03c0c95a0 2026-02-22T21:38:12.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.783+0000 7fe033fff640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe02400c760 tx=0x7fe02400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:12.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.783+0000 7fe0317fa640 10 client.?.objecter ms_handle_connect 0x7fe03c0c95a0 2026-02-22T21:38:12.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.783+0000 7fe0317fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:12.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0240052d0 con 0x7fe03c0c95a0 2026-02-22T21:38:12.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:12.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: dump: 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: _finish_auth 0 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: _check_auth_tickets 2026-02-22T21:38:12.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe024005490 con 0x7fe03c0c95a0 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02400a880 con 0x7fe03c0c95a0 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe032ffd640 4 set_mon_vals no callback set 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe0317fa640 10 monclient: dump: 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:12.790 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 5 monclient: authenticate success, global_id 4680 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _renew_subs 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03c1f2e50 con 0x7fe03c0c95a0 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _renew_subs 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03c1f3410 con 0x7fe03c0c95a0 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 1 librados: init done 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:12.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.785+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe03c0c5ab0 con 0x7fe03c0c95a0 2026-02-22T21:38:12.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.796+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe02400aa20 con 0x7fe03c0c95a0 2026-02-22T21:38:12.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.796+0000 7fe0317fa640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 0x7fe00c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:12.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.797+0000 7fe040e78640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 0x7fe00c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:12.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.797+0000 7fe040e78640 10 monclient: get_auth_request con 0x7fe00c078e50 auth_method 0 2026-02-22T21:38:12.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.797+0000 7fe040e78640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 0x7fe00c07b310 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe02c039000 tx=0x7fe02c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:12.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.797+0000 7fe0317fa640 10 client.4680.objecter ms_handle_connect 0x7fe00c078e50 2026-02-22T21:38:12.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.798+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe02400ad40 con 0x7fe03c0c95a0 2026-02-22T21:38:12.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.798+0000 7fe0317fa640 10 client.4680.objecter ms_dispatch2 0x7fe03c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:12.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.798+0000 7fe0317fa640 3 client.4680.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:12.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.798+0000 7fe0317fa640 3 client.4680.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:12.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.798+0000 7fe0317fa640 20 client.4680.objecter dump_active .. 0 homeless 2026-02-22T21:38:12.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.901+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe024062010 con 0x7fe03c0c95a0 2026-02-22T21:38:12.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.901+0000 7fe0317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:12.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:12.901+0000 7fe0317fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:13.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.001+0000 7fe043103640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:13.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.001+0000 7fe043103640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:13.003 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.001+0000 7fe043103640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:13.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.001+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe03c0c69e0 con 0x7fe03c0c95a0 2026-02-22T21:38:13.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe0317fa640 1 -- 10.20.193.51:0/3435100402 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe024066310 con 0x7fe03c0c95a0 2026-02-22T21:38:13.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe0317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:13.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe0317fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:13.006 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:13.006 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 10 librados: watch_flush enter 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 10 librados: watch_flush exit 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 20 client.4680.objecter shutdown clearing up homeless session... 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 10 client.4680.objecter successfully canceled tick 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 msgr2=0x7fe00c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 0x7fe00c07b310 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe02c039000 tx=0x7fe02c033000 comp rx=0 tx=0).stop 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 10 monclient: shutdown 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 msgr2=0x7fe03c1ef910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe02400c760 tx=0x7fe02400cc30 comp rx=0 tx=0).stop 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 shutdown_connections 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe00c078e50 0x7fe00c07b310 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:13.007 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe03c152ab0 0x7fe03c1efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe03c0c95a0 0x7fe03c1ef910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 --2- 10.20.193.51:0/3435100402 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe03c06d9d0 0x7fe03c15b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 >> 10.20.193.51:0/3435100402 conn(0x7fe03c0cd7b0 msgr2=0x7fe03c1516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.004+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 shutdown_connections 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.005+0000 7fe043103640 1 -- 10.20.193.51:0/3435100402 wait complete. 2026-02-22T21:38:13.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:13.005+0000 7fe043103640 1 librados: shutdown 2026-02-22T21:38:16.016 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.083+0000 7fc3a796f640 10 monclient: get_monmap_and_config 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.083+0000 7fc3a796f640 10 monclient: build_initial_monmap 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.083+0000 7fc3a796f640 10 monclient: monmap: 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:16.085601+0000 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:16.085601+0000 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:16.086 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.084+0000 7fc3a796f640 1 Processor -- start 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.084+0000 7fc3a796f640 1 -- start start 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.084+0000 7fc3a796f640 10 monclient: init 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient: _add_conns ranks=[3,1,0,5,2,4] 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc3a01563f0 0x7fc3a00cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a01563f0 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fc3a00c50e0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3a015f620 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a006d6d0 con 0x7fc3a015f620 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a0069210 con 0x7fc3a00c50e0 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:16.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a006f9c0 con 0x7fc3a01563f0 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a796f640 10 monclient(hunting): authenticate will time out at 1171.666337s 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a56e4640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc3a01563f0 0x7fc3a00cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a4ee3640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fc3a00c50e0 0x7fc3a006f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:43606/0 (socket says 10.20.193.51:43606) 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.085+0000 7fc3a4ee3640 1 -- 10.20.193.51:0/3945894850 learned_addr learned my addr 10.20.193.51:0/3945894850 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): get_auth_request con 0x7fc3a01563f0 auth_method 0 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:16.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.086+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 1 -- 10.20.193.51:0/3945894850 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216518929 0 0) 0x7fc3a0069210 con 0x7fc3a00c50e0 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 10 monclient(hunting): _negotiate 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 10 monclient(hunting): my global_id is 4669 2026-02-22T21:38:16.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.087+0000 7fc38ffff640 1 -- 10.20.193.51:0/3945894850 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc388003620 con 0x7fc3a00c50e0 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_done global_id 4683 payload 274 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 1 -- 10.20.193.51:0/3945894850 >> v1:10.20.193.68:6789/0 conn(0x7fc3a00c50e0 legacy=0x7fc3a006f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 1 -- 10.20.193.51:0/3945894850 >> v1:10.20.193.51:6789/0 conn(0x7fc3a015f620 legacy=0x7fc3a015fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 10 monclient: _finish_hunting 0 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 1 monclient: found mon.noname-a 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.088+0000 7fc3a56e4640 1 -- 10.20.193.51:0/3945894850 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a006e4f0 con 0x7fc3a01563f0 2026-02-22T21:38:16.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.090+0000 7fc3a56e4640 1 --2- 10.20.193.51:0/3945894850 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a01563f0 0x7fc3a00cefa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3900098a0 tx=0x7fc39002ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2447d47b3e81125 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 1 -- 10.20.193.51:0/3945894850 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39003c070 con 0x7fc3a01563f0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: dump: 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: _finish_auth 0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 1 -- 10.20.193.51:0/3945894850 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39002fb70 con 0x7fc3a01563f0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 1 -- 10.20.193.51:0/3945894850 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390037440 con 0x7fc3a01563f0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc3a796f640 5 monclient: authenticate success, global_id 4683 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc3a796f640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc3a796f640 4 set_mon_vals no callback set 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.091+0000 7fc38ffff640 10 monclient: dump: 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:16.094 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 10 monclient: shutdown 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 -- 10.20.193.51:0/3945894850 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a01563f0 msgr2=0x7fc3a00cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 --2- 10.20.193.51:0/3945894850 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a01563f0 0x7fc3a00cefa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3900098a0 tx=0x7fc39002ee60 comp rx=0 tx=0).stop 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 -- 10.20.193.51:0/3945894850 shutdown_connections 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 --2- 10.20.193.51:0/3945894850 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a01563f0 0x7fc3a00cefa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 -- 10.20.193.51:0/3945894850 >> 10.20.193.51:0/3945894850 conn(0x7fc3a00ce4b0 msgr2=0x7fc3a00ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 -- 10.20.193.51:0/3945894850 shutdown_connections 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 1 -- 10.20.193.51:0/3945894850 wait complete. 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 10 monclient: build_initial_monmap 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.092+0000 7fc3a796f640 10 monclient: monmap: 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:16.095 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 librados: starting msgr at 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 librados: starting objecter 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 Processor -- start 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 -- start start 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 librados: setting wanted keys 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 librados: calling monclient init 2026-02-22T21:38:16.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient: init 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc3a00c50e0 0x7fc3a00c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a00c50e0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3a01563f0 0x7fc3a00c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a01563f0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a015f620 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a006ee20 con 0x7fc3a015f620 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a01567d0 con 0x7fc3a00c50e0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a006eab0 con 0x7fc3a01563f0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a796f640 10 monclient(hunting): authenticate will time out at 1171.674696s 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.093+0000 7fc3a5ee5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc3a00c50e0 0x7fc3a00c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:50906/0 (socket says 10.20.193.51:50906) 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 1 -- 10.20.193.51:0/842430254 learned_addr learned my addr 10.20.193.51:0/842430254 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): get_auth_request con 0x7fc3a015f620 auth_method 0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): get_auth_request con 0x7fc3a00c50e0 auth_method 0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3a01563f0 0x7fc3a00c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:16.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a5ee5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): get_auth_request con 0x7fc3a01563f0 auth_method 0 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a56e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.094+0000 7fc3a4ee3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 10 monclient(hunting): handle_auth_done global_id 4689 payload 824 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 -- 10.20.193.51:0/842430254 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3a01563f0 msgr2=0x7fc3a00c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3a01563f0 0x7fc3a00c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 -- 10.20.193.51:0/842430254 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc3a00c50e0 msgr2=0x7fc3a00c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc3a00c50e0 0x7fc3a00c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 10 monclient: _finish_hunting 0 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 monclient: found mon.noname-a 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.095+0000 7fc3a5ee5640 1 -- 10.20.193.51:0/842430254 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a00c6ee0 con 0x7fc3a015f620 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.096+0000 7fc3a5ee5640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00cc50 tx=0x7fc39c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:16.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.096+0000 7fc38dffb640 10 client.?.objecter ms_handle_connect 0x7fc3a015f620 2026-02-22T21:38:16.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.096+0000 7fc38dffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c003a70 con 0x7fc3a015f620 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: dump: 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: _finish_auth 0 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39c003c10 con 0x7fc3a015f620 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 5 monclient: authenticate success, global_id 4689 2026-02-22T21:38:16.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _renew_subs 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a00c71c0 con 0x7fc3a015f620 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0050d0 con 0x7fc3a015f620 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38f7fe640 4 set_mon_vals no callback set 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc38dffb640 10 monclient: dump: 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _renew_subs 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a00c7780 con 0x7fc3a015f620 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 1 librados: init done 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.098+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a0068e90 con 0x7fc3a015f620 2026-02-22T21:38:16.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.109+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc39c005270 con 0x7fc3a015f620 2026-02-22T21:38:16.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.109+0000 7fc38dffb640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 0x7fc37007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:16.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.109+0000 7fc3a56e4640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 0x7fc37007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:16.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.109+0000 7fc3a56e4640 10 monclient: get_auth_request con 0x7fc370078e50 auth_method 0 2026-02-22T21:38:16.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.110+0000 7fc3a56e4640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 0x7fc37007b310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc390039000 tx=0x7fc390033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:16.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.110+0000 7fc38dffb640 10 client.4689.objecter ms_handle_connect 0x7fc370078e50 2026-02-22T21:38:16.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.111+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc39c05e770 con 0x7fc3a015f620 2026-02-22T21:38:16.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.111+0000 7fc38dffb640 10 client.4689.objecter ms_dispatch2 0x7fc3a0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:16.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.111+0000 7fc38dffb640 3 client.4689.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:16.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.111+0000 7fc38dffb640 3 client.4689.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:16.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.111+0000 7fc38dffb640 20 client.4689.objecter dump_active .. 0 homeless 2026-02-22T21:38:16.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.214+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc39c062730 con 0x7fc3a015f620 2026-02-22T21:38:16.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.214+0000 7fc38dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:16.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.214+0000 7fc38dffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:16.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.316+0000 7fc3a796f640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:16.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.316+0000 7fc3a796f640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:16.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.316+0000 7fc3a796f640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:16.319 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.316+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc3a00c4ec0 con 0x7fc3a015f620 2026-02-22T21:38:16.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.319+0000 7fc38dffb640 1 -- 10.20.193.51:0/842430254 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fc39c066a30 con 0x7fc3a015f620 2026-02-22T21:38:16.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.319+0000 7fc38dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:16.321 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.319+0000 7fc38dffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:16.321 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:16.321 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 10 librados: watch_flush enter 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 10 librados: watch_flush exit 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 20 client.4689.objecter shutdown clearing up homeless session... 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 10 client.4689.objecter successfully canceled tick 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 msgr2=0x7fc37007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 0x7fc37007b310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc390039000 tx=0x7fc390033000 comp rx=0 tx=0).stop 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 10 monclient: shutdown 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:16.323 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 msgr2=0x7fc3a00c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00cc50 tx=0x7fc39c00a430 comp rx=0 tx=0).stop 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 shutdown_connections 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc370078e50 0x7fc37007b310 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc3a015f620 0x7fc3a00c9370 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc3a01563f0 0x7fc3a00c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 --2- 10.20.193.51:0/842430254 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc3a00c50e0 0x7fc3a00c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 >> 10.20.193.51:0/842430254 conn(0x7fc3a00ce4b0 msgr2=0x7fc3a015fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 shutdown_connections 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 -- 10.20.193.51:0/842430254 wait complete. 2026-02-22T21:38:16.324 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:16.320+0000 7fc3a796f640 1 librados: shutdown 2026-02-22T21:38:19.332 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 10 monclient: get_monmap_and_config 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 10 monclient: build_initial_monmap 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 10 monclient: monmap: 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:19.399820+0000 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:19.399820+0000 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.400 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 1 Processor -- start 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 1 -- start start 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.398+0000 7f47cfe16640 10 monclient: init 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient: _add_conns ranks=[3,0,5,1,2,4] 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 0x7f47c80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-a con 0x7f47c8150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47c8150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f47c815c710 0x7f47c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-c con 0x7f47c815c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c815a920 con 0x7f47c8150600 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f47c815af60 con 0x7f47c8150220 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f47c815b240 con 0x7f47c815c710 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:19.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cfe16640 10 monclient(hunting): authenticate will time out at 1174.980166s 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 0x7f47c80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47ce38c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f47c815c710 0x7f47c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 0x7f47c80c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35696/0 (socket says 10.20.193.51:35696) 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 1 -- 10.20.193.51:0/3905911598 learned_addr learned my addr 10.20.193.51:0/3905911598 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 10 monclient(hunting): get_auth_request con 0x7f47c8150220 auth_method 0 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:19.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47cdb8b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47ce38c640 10 monclient(hunting): get_auth_request con 0x7f47c815c710 auth_method 0 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47ce38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47ce38c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.399+0000 7f47ce38c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.400+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 10 monclient(hunting): handle_auth_done global_id 4634 payload 274 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 1 -- 10.20.193.51:0/3905911598 >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 msgr2=0x7f47c80c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 1 --2- 10.20.193.51:0/3905911598 >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 0x7f47c80c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 1 -- 10.20.193.51:0/3905911598 >> v1:10.20.193.51:6789/0 conn(0x7f47c8150600 legacy=0x7f47c815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 10 monclient: _finish_hunting 0 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 1 monclient: found mon.noname-c 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.401+0000 7f47ce38c640 1 -- 10.20.193.51:0/3905911598 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c815b520 con 0x7f47c815c710 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.402+0000 7f47ce38c640 1 --2- 10.20.193.51:0/3905911598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c815c710 0x7f47c806fca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47c80c65c0 tx=0x7f47c402f250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cca48e03dc407b8f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:19.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 1 -- 10.20.193.51:0/3905911598 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c403c070 con 0x7f47c815c710 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: dump: 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: _finish_auth 0 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: _check_auth_tickets 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 1 -- 10.20.193.51:0/3905911598 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c4037440 con 0x7f47c815c710 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47cfe16640 5 monclient: authenticate success, global_id 4634 2026-02-22T21:38:19.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47cfe16640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 1 -- 10.20.193.51:0/3905911598 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4037810 con 0x7f47c815c710 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47ccb89640 10 monclient: dump: 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47cfe16640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.403+0000 7f47cfe16640 4 set_mon_vals no callback set 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 10 monclient: shutdown 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- 10.20.193.51:0/3905911598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c815c710 msgr2=0x7f47c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 --2- 10.20.193.51:0/3905911598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c815c710 0x7f47c806fca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47c80c65c0 tx=0x7f47c402f250 comp rx=0 tx=0).stop 2026-02-22T21:38:19.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- 10.20.193.51:0/3905911598 shutdown_connections 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 --2- 10.20.193.51:0/3905911598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c815c710 0x7f47c806fca0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 --2- 10.20.193.51:0/3905911598 >> v2:10.20.193.51:3300/0 conn(0x7f47c8150220 0x7f47c80c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- 10.20.193.51:0/3905911598 >> 10.20.193.51:0/3905911598 conn(0x7f47c80cd7b0 msgr2=0x7f47c80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- 10.20.193.51:0/3905911598 shutdown_connections 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- 10.20.193.51:0/3905911598 wait complete. 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 10 monclient: build_initial_monmap 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 10 monclient: monmap: 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 librados: starting msgr at 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 librados: starting objecter 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 Processor -- start 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 -- start start 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 librados: setting wanted keys 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 1 librados: calling monclient init 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.404+0000 7f47cfe16640 10 monclient: init 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:38:19.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c8150220 0x7f47c81e6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-c con 0x7f47c8150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f47c8150600 0x7f47c814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-b con 0x7f47c8150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): picked mon.noname-a con 0x7f47c815c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f47c80701e0 con 0x7f47c815c710 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f47c815bae0 con 0x7f47c8150600 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f47c8070360 con 0x7f47c8150220 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cfe16640 10 monclient(hunting): authenticate will time out at 1174.986497s 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cdb8b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c8150220 0x7f47c81e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35702/0 (socket says 10.20.193.51:35702) 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 1 -- 10.20.193.51:0/440859353 learned_addr learned my addr 10.20.193.51:0/440859353 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 10 monclient(hunting): get_auth_request con 0x7f47c815c710 auth_method 0 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:19.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47ce38c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cdb8b640 10 monclient(hunting): get_auth_request con 0x7f47c8150220 auth_method 0 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cdb8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cdb8b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.405+0000 7f47cdb8b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f47c8150600 0x7f47c814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47ce38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cdb8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): get_auth_request con 0x7f47c8150600 auth_method 0 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:19.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.406+0000 7f47cd38a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 10 monclient(hunting): handle_auth_done global_id 4698 payload 824 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 -- 10.20.193.51:0/440859353 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c8150220 msgr2=0x7f47c81e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c8150220 0x7f47c81e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 -- 10.20.193.51:0/440859353 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f47c8150600 msgr2=0x7f47c814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f47c8150600 0x7f47c814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 10 monclient: _finish_hunting 0 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 monclient: found mon.noname-a 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.407+0000 7f47ce38c640 1 -- 10.20.193.51:0/440859353 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c81e5040 con 0x7f47c815c710 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.408+0000 7f47ce38c640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f47c402f780 tx=0x7f47c4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.408+0000 7f47b67fc640 10 client.?.objecter ms_handle_connect 0x7f47c815c710 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.408+0000 7f47b67fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c403c050 con 0x7f47c815c710 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: dump: 2026-02-22T21:38:19.412 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: _finish_auth 0 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 5 monclient: authenticate success, global_id 4698 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c4048070 con 0x7f47c815c710 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4035a90 con 0x7f47c815c710 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _renew_subs 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b7fff640 4 set_mon_vals no callback set 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c81e52b0 con 0x7f47c815c710 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47b67fc640 10 monclient: dump: 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _renew_subs 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c81e5b50 con 0x7f47c815c710 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 1 librados: init done 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:19.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:19.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:19.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.410+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c80c46c0 con 0x7f47c815c710 2026-02-22T21:38:19.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.422+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f47c4035c30 con 0x7f47c815c710 2026-02-22T21:38:19.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.422+0000 7f47b67fc640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 0x7f479807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:19.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.422+0000 7f47cdb8b640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 0x7f479807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:19.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.422+0000 7f47cdb8b640 10 monclient: get_auth_request con 0x7f4798078e50 auth_method 0 2026-02-22T21:38:19.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.423+0000 7f47b67fc640 10 client.4698.objecter ms_handle_connect 0x7f4798078e50 2026-02-22T21:38:19.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.423+0000 7f47cdb8b640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 0x7f479807b310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006dc0 tx=0x7f47b8009340 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:19.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f47c40845a0 con 0x7f47c815c710 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 10 client.4698.objecter ms_dispatch2 0x7f47c8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 3 client.4698.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 3 client.4698.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 20 client.4698.objecter dump_active .. 0 homeless 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f47c80c46c0 con 0x7f47c815c710 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:19.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.523+0000 7f47b67fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:19.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.625+0000 7f47cfe16640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:19.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.625+0000 7f47cfe16640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:19.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.625+0000 7f47cfe16640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:19.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.625+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f47c806fca0 con 0x7f47c815c710 2026-02-22T21:38:19.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.627+0000 7f47b67fc640 1 -- 10.20.193.51:0/440859353 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f47c4088560 con 0x7f47c815c710 2026-02-22T21:38:19.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.627+0000 7f47b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:19.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.627+0000 7f47b67fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:19.630 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:19.630 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:19.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 10 librados: watch_flush enter 2026-02-22T21:38:19.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 10 librados: watch_flush exit 2026-02-22T21:38:19.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 20 client.4698.objecter shutdown clearing up homeless session... 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 10 client.4698.objecter successfully canceled tick 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 msgr2=0x7f479807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 0x7f479807b310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006dc0 tx=0x7f47b8009340 comp rx=0 tx=0).stop 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 10 monclient: shutdown 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 msgr2=0x7f47c80c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f47c402f780 tx=0x7f47c4004b40 comp rx=0 tx=0).stop 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 shutdown_connections 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4798078e50 0x7f479807b310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f47c815c710 0x7f47c80c10d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f47c8150600 0x7f47c814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 --2- 10.20.193.51:0/440859353 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f47c8150220 0x7f47c81e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:19.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.628+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 >> 10.20.193.51:0/440859353 conn(0x7f47c80cd7b0 msgr2=0x7f47c806eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:19.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.629+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 shutdown_connections 2026-02-22T21:38:19.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.629+0000 7f47cfe16640 1 -- 10.20.193.51:0/440859353 wait complete. 2026-02-22T21:38:19.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:19.629+0000 7f47cfe16640 1 librados: shutdown 2026-02-22T21:38:21.344 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:52.497 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.345 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:52.499 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.345 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:52.497 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.347 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:52.499 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.353 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:02:52.507 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.356 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:02:52.507 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.374 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:52.490 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.375 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:52.491 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:21.384 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:02:52.500 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:22.641 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:22.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.707+0000 7f89d992c640 10 monclient: get_monmap_and_config 2026-02-22T21:38:22.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.707+0000 7f89d992c640 10 monclient: build_initial_monmap 2026-02-22T21:38:22.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.707+0000 7f89d992c640 10 monclient: monmap: 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:22.709525+0000 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:22.709525+0000 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:22.711 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:22.712 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:22.712 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 Processor -- start 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 -- start start 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient: init 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient: _add_conns ranks=[4,2,0,3,1,5] 2026-02-22T21:38:22.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f89d4152ab0 0x7f89d40c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-b con 0x7f89d4152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89d406d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89d40c95a0 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d406f1d0 con 0x7f89d40c95a0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d406de50 con 0x7f89d406d9d0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f89d406ed40 con 0x7f89d4152ab0 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.708+0000 7f89d992c640 10 monclient(hunting): authenticate will time out at 1178.289983s 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d37fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f89d40c95a0 0x7f89d4158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:35870/0 (socket says 10.20.193.51:35870) 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d37fe640 1 -- 10.20.193.51:0/3857782449 learned_addr learned my addr 10.20.193.51:0/3857782449 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:22.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d2ffd640 1 --2- 10.20.193.51:0/3857782449 >> v2:10.20.193.68:3300/0 conn(0x7f89d4152ab0 0x7f89d40c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d2ffd640 10 monclient(hunting): get_auth_request con 0x7f89d4152ab0 auth_method 0 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.709+0000 7f89d2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386343294 0 0) 0x7f89d406f1d0 con 0x7f89d40c95a0 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): my global_id is 4701 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c0003620 con 0x7f89d40c95a0 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744995144 0 0) 0x7f89d406de50 con 0x7f89d406d9d0 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 10 monclient(hunting): my global_id is 4640 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.710+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d406f1d0 con 0x7f89d406d9d0 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_done global_id 4678 payload 274 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 1 -- 10.20.193.51:0/3857782449 >> v1:10.20.193.125:6789/0 conn(0x7f89d406d9d0 legacy=0x7f89d40c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 1 -- 10.20.193.51:0/3857782449 >> v1:10.20.193.51:6789/0 conn(0x7f89d40c95a0 legacy=0x7f89d4158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:38:22.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 1 monclient: found mon.noname-b 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 1 -- 10.20.193.51:0/3857782449 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d406e130 con 0x7f89d4152ab0 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.711+0000 7f89d2ffd640 1 --2- 10.20.193.51:0/3857782449 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d4152ab0 0x7f89d40c5ab0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f89c80098d0 tx=0x7f89c802ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58533fde21c0b304 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c803c070 con 0x7f89d4152ab0 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: dump: 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.725 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c802fbf0 con 0x7f89d4152ab0 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d992c640 5 monclient: authenticate success, global_id 4678 2026-02-22T21:38:22.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 1 -- 10.20.193.51:0/3857782449 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8037440 con 0x7f89d4152ab0 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d992c640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d992c640 4 set_mon_vals no callback set 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.713+0000 7f89d1ffb640 10 monclient: dump: 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:22.727 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 10 monclient: shutdown 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- 10.20.193.51:0/3857782449 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d4152ab0 msgr2=0x7f89d40c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 --2- 10.20.193.51:0/3857782449 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d4152ab0 0x7f89d40c5ab0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f89c80098d0 tx=0x7f89c802ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- 10.20.193.51:0/3857782449 shutdown_connections 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 --2- 10.20.193.51:0/3857782449 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d4152ab0 0x7f89d40c5ab0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- 10.20.193.51:0/3857782449 >> 10.20.193.51:0/3857782449 conn(0x7f89d40cd7b0 msgr2=0x7f89d40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:22.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- 10.20.193.51:0/3857782449 shutdown_connections 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- 10.20.193.51:0/3857782449 wait complete. 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 10 monclient: build_initial_monmap 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 10 monclient: monmap: 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:22.729 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 librados: starting msgr at 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 librados: starting objecter 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 Processor -- start 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 -- start start 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 librados: setting wanted keys 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 1 librados: calling monclient init 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.714+0000 7f89d992c640 10 monclient: init 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f89d406d9d0 0x7f89d40cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:22.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-c con 0x7f89d406d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-b con 0x7f89d40c95a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): picked mon.noname-a con 0x7f89d4152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f89d406e9d0 con 0x7f89d4152ab0 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f89d406e6f0 con 0x7f89d40c95a0 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4152e90 con 0x7f89d406d9d0 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d992c640 10 monclient(hunting): authenticate will time out at 1178.296397s 2026-02-22T21:38:22.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d2ffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f89d406d9d0 0x7f89d40cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35724/0 (socket says 10.20.193.51:35724) 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 1 -- 10.20.193.51:0/3916534925 learned_addr learned my addr 10.20.193.51:0/3916534925 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35708/0 (socket says 10.20.193.51:35708) 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 10 monclient(hunting): get_auth_request con 0x7f89d40c95a0 auth_method 0 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d27fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 10 monclient(hunting): get_auth_request con 0x7f89d4152ab0 auth_method 0 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:22.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.715+0000 7f89d37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): get_auth_request con 0x7f89d406d9d0 auth_method 0 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:22.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 msgr2=0x7f89d414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 msgr2=0x7f89d414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.716+0000 7f89d27fc640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 10 monclient(hunting): handle_auth_done global_id 4704 payload 824 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f89d406d9d0 msgr2=0x7f89d40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f89d406d9d0 0x7f89d40cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 msgr2=0x7f89d414afd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 monclient: found mon.noname-a 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.717+0000 7f89d37fe640 1 -- 10.20.193.51:0/3916534925 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d41ef5f0 con 0x7f89d4152ab0 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.718+0000 7f89d37fe640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f89c400ca80 tx=0x7f89c400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.718+0000 7f89abfff640 10 client.?.objecter ms_handle_connect 0x7f89d4152ab0 2026-02-22T21:38:22.735 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.718+0000 7f89abfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:22.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c400ac40 con 0x7f89d4152ab0 2026-02-22T21:38:22.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:22.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: dump: 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.739 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: _finish_auth 0 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: _check_auth_tickets 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:22.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c400ade0 con 0x7f89d4152ab0 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 5 monclient: authenticate success, global_id 4704 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _renew_subs 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d41ef8d0 con 0x7f89d4152ab0 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4012600 con 0x7f89d4152ab0 2026-02-22T21:38:22.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d17fa640 4 set_mon_vals no callback set 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89abfff640 10 monclient: dump: 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:22.742 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 client.4704.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _renew_subs 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d41efe90 con 0x7f89d4152ab0 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 1 librados: init done 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:22.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.720+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d40c5ab0 con 0x7f89d4152ab0 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.732+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f89c40127a0 con 0x7f89d4152ab0 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.732+0000 7f89abfff640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 0x7f89a407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.733+0000 7f89d2ffd640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 0x7f89a407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.733+0000 7f89d2ffd640 10 monclient: get_auth_request con 0x7f89a4078e00 auth_method 0 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.733+0000 7f89abfff640 10 client.4704.objecter ms_handle_connect 0x7f89a4078e00 2026-02-22T21:38:22.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.733+0000 7f89d2ffd640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 0x7f89a407b2c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f89c8039000 tx=0x7f89c8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f89c405e810 con 0x7f89d4152ab0 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 10 client.4704.objecter ms_dispatch2 0x7f89d4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 3 client.4704.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 3 client.4704.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 20 client.4704.objecter dump_active .. 0 homeless 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f89c40627d0 con 0x7f89d4152ab0 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:22.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.838+0000 7f89abfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:22.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.938+0000 7f89d992c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:22.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.938+0000 7f89d992c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:22.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.938+0000 7f89d992c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:22.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.938+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f89d40c8910 con 0x7f89d4152ab0 2026-02-22T21:38:22.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.940+0000 7f89abfff640 1 -- 10.20.193.51:0/3916534925 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f89c4066ad0 con 0x7f89d4152ab0 2026-02-22T21:38:22.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.940+0000 7f89abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:22.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.940+0000 7f89abfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:22.944 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:22.944 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 10 librados: watch_flush enter 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 10 librados: watch_flush exit 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 20 client.4704.objecter shutdown clearing up homeless session... 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 10 client.4704.objecter successfully canceled tick 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 msgr2=0x7f89a407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 0x7f89a407b2c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f89c8039000 tx=0x7f89c8033000 comp rx=0 tx=0).stop 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 10 monclient: shutdown 2026-02-22T21:38:22.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 msgr2=0x7f89d414b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f89c400ca80 tx=0x7f89c400cf50 comp rx=0 tx=0).stop 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 shutdown_connections 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f89a4078e00 0x7f89a407b2c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f89d4152ab0 0x7f89d414b510 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f89d40c95a0 0x7f89d414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 --2- 10.20.193.51:0/3916534925 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f89d406d9d0 0x7f89d40cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 >> 10.20.193.51:0/3916534925 conn(0x7f89d40cd7b0 msgr2=0x7f89d40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 shutdown_connections 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 -- 10.20.193.51:0/3916534925 wait complete. 2026-02-22T21:38:22.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:22.942+0000 7f89d992c640 1 librados: shutdown 2026-02-22T21:38:25.954 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 10 monclient: get_monmap_and_config 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 10 monclient: build_initial_monmap 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 10 monclient: monmap: 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:26.019877+0000 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:26.019877+0000 2026-02-22T21:38:26.021 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 1 Processor -- start 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 1 -- start start 2026-02-22T21:38:26.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.019+0000 7fad293c3640 10 monclient: init 2026-02-22T21:38:26.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:26.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient: _add_conns ranks=[4,1,2,3,5,0] 2026-02-22T21:38:26.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fad24152ab0 0x7fad240c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:26.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-b con 0x7fad24152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fad2406d9d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad240c95a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2406f1d0 con 0x7fad2406d9d0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad2406de50 con 0x7fad240c95a0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fad2406ed40 con 0x7fad24152ab0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad293c3640 10 monclient(hunting): authenticate will time out at 1181.600196s 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad22ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fad24152ab0 0x7fad240c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad227fc640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fad2406d9d0 0x7fad240c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:37416/0 (socket says 10.20.193.51:37416) 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad227fc640 1 -- 10.20.193.51:0/2688091830 learned_addr learned my addr 10.20.193.51:0/2688091830 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:26.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad22ffd640 10 monclient(hunting): get_auth_request con 0x7fad24152ab0 auth_method 0 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad22ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.020+0000 7fad22ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.021+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.021+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.021+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 1 -- 10.20.193.51:0/2688091830 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489761867 0 0) 0x7fad2406de50 con 0x7fad240c95a0 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 10 monclient(hunting): my global_id is 4646 2026-02-22T21:38:26.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad21ffb640 1 -- 10.20.193.51:0/2688091830 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad100058b0 con 0x7fad240c95a0 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 10 monclient(hunting): handle_auth_done global_id 4684 payload 274 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 1 -- 10.20.193.51:0/2688091830 >> v1:10.20.193.125:6789/0 conn(0x7fad240c95a0 legacy=0x7fad24158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 1 -- 10.20.193.51:0/2688091830 >> v1:10.20.193.68:6789/0 conn(0x7fad2406d9d0 legacy=0x7fad240c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 1 monclient: found mon.noname-b 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.022+0000 7fad22ffd640 1 -- 10.20.193.51:0/2688091830 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2406e130 con 0x7fad24152ab0 2026-02-22T21:38:26.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.024+0000 7fad22ffd640 1 --2- 10.20.193.51:0/2688091830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad24152ab0 0x7fad240c5ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad180098d0 tx=0x7fad1802ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d65bbc96e82c154 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:26.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 1 -- 10.20.193.51:0/2688091830 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1803c070 con 0x7fad24152ab0 2026-02-22T21:38:26.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:26.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: dump: 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: _finish_auth 0 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 1 -- 10.20.193.51:0/2688091830 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1802fbf0 con 0x7fad24152ab0 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad293c3640 5 monclient: authenticate success, global_id 4684 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad293c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 1 -- 10.20.193.51:0/2688091830 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18037440 con 0x7fad24152ab0 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad21ffb640 10 monclient: dump: 2026-02-22T21:38:26.037 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad293c3640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad293c3640 4 set_mon_vals no callback set 2026-02-22T21:38:26.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.026+0000 7fad293c3640 10 monclient: shutdown 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- 10.20.193.51:0/2688091830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad24152ab0 msgr2=0x7fad240c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 --2- 10.20.193.51:0/2688091830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad24152ab0 0x7fad240c5ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fad180098d0 tx=0x7fad1802ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- 10.20.193.51:0/2688091830 shutdown_connections 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 --2- 10.20.193.51:0/2688091830 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad24152ab0 0x7fad240c5ab0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- 10.20.193.51:0/2688091830 >> 10.20.193.51:0/2688091830 conn(0x7fad240cd7b0 msgr2=0x7fad240cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- 10.20.193.51:0/2688091830 shutdown_connections 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- 10.20.193.51:0/2688091830 wait complete. 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 10 monclient: build_initial_monmap 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 10 monclient: monmap: 2026-02-22T21:38:26.039 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:26.040 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 librados: starting msgr at 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 librados: starting objecter 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 Processor -- start 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 -- start start 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 librados: setting wanted keys 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 1 librados: calling monclient init 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.027+0000 7fad293c3640 10 monclient: init 2026-02-22T21:38:26.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-b con 0x7fad2406d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad240c95a0 0x7fad240c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-a con 0x7fad240c95a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad24152ab0 0x7fad24159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): picked mon.noname-c con 0x7fad24152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:26.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fad2406e9d0 con 0x7fad240c95a0 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fad2406e6f0 con 0x7fad2406d9d0 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fad24152e90 con 0x7fad24152ab0 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad293c3640 10 monclient(hunting): authenticate will time out at 1181.608249s 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad237fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad24152ab0 0x7fad24159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:26.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad227fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad240c95a0 0x7fad240c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:35742/0 (socket says 10.20.193.51:35742) 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 1 -- 10.20.193.51:0/3690476373 learned_addr learned my addr 10.20.193.51:0/3690476373 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): get_auth_request con 0x7fad2406d9d0 auth_method 0 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad237fe640 10 monclient(hunting): get_auth_request con 0x7fad24152ab0 auth_method 0 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad237fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad237fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad227fc640 10 monclient(hunting): get_auth_request con 0x7fad240c95a0 auth_method 0 2026-02-22T21:38:26.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad227fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad227fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.028+0000 7fad22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.029+0000 7fad227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 10 monclient(hunting): handle_auth_done global_id 4690 payload 824 2026-02-22T21:38:26.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 -- 10.20.193.51:0/3690476373 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad24152ab0 msgr2=0x7fad24159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad24152ab0 0x7fad24159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 -- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad240c95a0 msgr2=0x7fad240c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad240c95a0 0x7fad240c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 monclient: found mon.noname-b 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.030+0000 7fad22ffd640 1 -- 10.20.193.51:0/3690476373 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2415a8d0 con 0x7fad2406d9d0 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.031+0000 7fad22ffd640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18009130 tx=0x7fad180047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.031+0000 7fad037fe640 10 client.?.objecter ms_handle_connect 0x7fad2406d9d0 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.031+0000 7fad037fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:26.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18047070 con 0x7fad2406d9d0 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: dump: 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:26.047 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad18042630 con 0x7fad2406d9d0 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 5 monclient: authenticate success, global_id 4690 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _renew_subs 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2415abb0 con 0x7fad2406d9d0 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:26.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1803c040 con 0x7fad2406d9d0 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad037fe640 10 monclient: dump: 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:26.049 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad217fa640 4 set_mon_vals no callback set 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _renew_subs 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad240bf5e0 con 0x7fad2406d9d0 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 1 librados: init done 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.033+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad240c5ab0 con 0x7fad2406d9d0 2026-02-22T21:38:26.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fad1804b3b0 con 0x7fad2406d9d0 2026-02-22T21:38:26.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad037fe640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 0x7facf007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:26.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad227fc640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 0x7facf007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:26.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad227fc640 10 monclient: get_auth_request con 0x7facf0078e00 auth_method 0 2026-02-22T21:38:26.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad227fc640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 0x7facf007b2c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad240c4390 tx=0x7fad0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:26.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.142+0000 7fad037fe640 10 client.4690.objecter ms_handle_connect 0x7facf0078e00 2026-02-22T21:38:26.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.143+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fad1802fb50 con 0x7fad2406d9d0 2026-02-22T21:38:26.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.143+0000 7fad037fe640 10 client.4690.objecter ms_dispatch2 0x7fad24000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:26.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.143+0000 7fad037fe640 3 client.4690.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:26.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.143+0000 7fad037fe640 3 client.4690.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:26.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.143+0000 7fad037fe640 20 client.4690.objecter dump_active .. 0 homeless 2026-02-22T21:38:26.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.149+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fad18086a80 con 0x7fad2406d9d0 2026-02-22T21:38:26.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.149+0000 7fad037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:26.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.149+0000 7fad037fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:26.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.248+0000 7fad293c3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:26.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.248+0000 7fad293c3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:26.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.248+0000 7fad293c3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:26.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.248+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fad2414d380 con 0x7fad2406d9d0 2026-02-22T21:38:26.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.251+0000 7fad037fe640 1 -- 10.20.193.51:0/3690476373 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fad1808ad80 con 0x7fad2406d9d0 2026-02-22T21:38:26.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.251+0000 7fad037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:26.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.251+0000 7fad037fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:26.253 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:26.253 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:26.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 10 librados: watch_flush enter 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 10 librados: watch_flush exit 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 20 client.4690.objecter shutdown clearing up homeless session... 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 10 client.4690.objecter successfully canceled tick 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 msgr2=0x7facf007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 0x7facf007b2c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad240c4390 tx=0x7fad0c009040 comp rx=0 tx=0).stop 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 10 monclient: shutdown 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 msgr2=0x7fad2415b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:26.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18009130 tx=0x7fad180047c0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 shutdown_connections 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7facf0078e00 0x7facf007b2c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fad24152ab0 0x7fad24159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fad240c95a0 0x7fad240c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 --2- 10.20.193.51:0/3690476373 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fad2406d9d0 0x7fad2415b5e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 >> 10.20.193.51:0/3690476373 conn(0x7fad240cd7b0 msgr2=0x7fad240cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 shutdown_connections 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 -- 10.20.193.51:0/3690476373 wait complete. 2026-02-22T21:38:26.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:26.252+0000 7fad293c3640 1 librados: shutdown 2026-02-22T21:38:29.263 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.331+0000 7fb6090da640 10 monclient: get_monmap_and_config 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.331+0000 7fb6090da640 10 monclient: build_initial_monmap 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.331+0000 7fb6090da640 10 monclient: monmap: 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:29.332486+0000 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:29.332486+0000 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:29.336 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 Processor -- start 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 -- start start 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient: init 2026-02-22T21:38:29.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient: _add_conns ranks=[0,2,3,1,4,5] 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb604150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb60415c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb6040703b0 0x7fb6040c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-a con 0x7fb6040703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60415a920 con 0x7fb604150220 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60415af60 con 0x7fb60415c710 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fb60415b240 con 0x7fb6040703b0 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.332+0000 7fb6090da640 10 monclient(hunting): authenticate will time out at 1184.912909s 2026-02-22T21:38:29.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb603577640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb6040703b0 0x7fb6040c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb602d76640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fb604150220 0x7fb60415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:34524/0 (socket says 10.20.193.51:34524) 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb602d76640 1 -- 10.20.193.51:0/3730589185 learned_addr learned my addr 10.20.193.51:0/3730589185 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb603577640 10 monclient(hunting): get_auth_request con 0x7fb6040703b0 auth_method 0 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb603577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb603577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.333+0000 7fb603577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.334+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.334+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:29.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.334+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 1 -- 10.20.193.51:0/3730589185 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609414098 0 0) 0x7fb60415af60 con 0x7fb60415c710 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 10 monclient(hunting): _negotiate 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 10 monclient(hunting): my global_id is 4652 2026-02-22T21:38:29.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb601d74640 1 -- 10.20.193.51:0/3730589185 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5f0003620 con 0x7fb60415c710 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 10 monclient(hunting): handle_auth_done global_id 4710 payload 274 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 1 -- 10.20.193.51:0/3730589185 >> v1:10.20.193.125:6789/0 conn(0x7fb60415c710 legacy=0x7fb60406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 1 -- 10.20.193.51:0/3730589185 >> v1:10.20.193.51:6789/0 conn(0x7fb604150220 legacy=0x7fb60415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 10 monclient: _finish_hunting 0 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 1 monclient: found mon.noname-a 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.335+0000 7fb603577640 1 -- 10.20.193.51:0/3730589185 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60415b520 con 0x7fb6040703b0 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.337+0000 7fb603577640 1 --2- 10.20.193.51:0/3730589185 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6040c73a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009dc0 tx=0x7fb5f402f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c189349b293b66 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 1 -- 10.20.193.51:0/3730589185 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f403c070 con 0x7fb6040703b0 2026-02-22T21:38:29.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: dump: 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:29.347 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: _finish_auth 0 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: _check_auth_tickets 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 1 -- 10.20.193.51:0/3730589185 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f4037440 con 0x7fb6040703b0 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 5 monclient: authenticate success, global_id 4710 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 1 -- 10.20.193.51:0/3730589185 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4037810 con 0x7fb6040703b0 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 4 set_mon_vals no callback set 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:29.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb601d74640 10 monclient: dump: 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 10 monclient: shutdown 2026-02-22T21:38:29.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 -- 10.20.193.51:0/3730589185 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 msgr2=0x7fb6040c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 --2- 10.20.193.51:0/3730589185 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6040c73a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009dc0 tx=0x7fb5f402f2f0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 -- 10.20.193.51:0/3730589185 shutdown_connections 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 --2- 10.20.193.51:0/3730589185 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6040c73a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 -- 10.20.193.51:0/3730589185 >> 10.20.193.51:0/3730589185 conn(0x7fb6040cd7b0 msgr2=0x7fb6040cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 -- 10.20.193.51:0/3730589185 shutdown_connections 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.339+0000 7fb6090da640 1 -- 10.20.193.51:0/3730589185 wait complete. 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient: build_initial_monmap 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient: monmap: 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:29.350 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 librados: starting msgr at 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 librados: starting objecter 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 Processor -- start 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 -- start start 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 librados: setting wanted keys 2026-02-22T21:38:29.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 librados: calling monclient init 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient: init 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-a con 0x7fb6040703b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb604150220 0x7fb6040bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-b con 0x7fb604150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb60415c710 0x7fb6041e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): picked mon.noname-c con 0x7fb60415c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb604150600 con 0x7fb6040703b0 2026-02-22T21:38:29.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb60415bae0 con 0x7fb604150220 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb604150780 con 0x7fb60415c710 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.340+0000 7fb6090da640 10 monclient(hunting): authenticate will time out at 1184.920787s 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:45848/0 (socket says 10.20.193.51:45848) 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 1 -- 10.20.193.51:0/1809073413 learned_addr learned my addr 10.20.193.51:0/1809073413 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): get_auth_request con 0x7fb6040703b0 auth_method 0 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:29.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb60415c710 0x7fb6041e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb604150220 0x7fb6040bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): get_auth_request con 0x7fb60415c710 auth_method 0 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): get_auth_request con 0x7fb604150220 auth_method 0 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:29.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb603577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.341+0000 7fb602575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 10 monclient(hunting): handle_auth_done global_id 4716 payload 824 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 -- 10.20.193.51:0/1809073413 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb60415c710 msgr2=0x7fb6041e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb60415c710 0x7fb6041e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 -- 10.20.193.51:0/1809073413 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb604150220 msgr2=0x7fb6040bfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb604150220 0x7fb6040bfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 10 monclient: _finish_hunting 0 2026-02-22T21:38:29.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 monclient: found mon.noname-a 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.342+0000 7fb602d76640 1 -- 10.20.193.51:0/1809073413 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6040c0170 con 0x7fb6040703b0 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.343+0000 7fb602d76640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800c700 tx=0x7fb5f800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.343+0000 7fb5e37fe640 10 client.?.objecter ms_handle_connect 0x7fb6040703b0 2026-02-22T21:38:29.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.343+0000 7fb5e37fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:29.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8017070 con 0x7fb6040703b0 2026-02-22T21:38:29.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:29.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:29.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: dump: 2026-02-22T21:38:29.358 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:29.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f8005bb0 con 0x7fb6040703b0 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 5 monclient: authenticate success, global_id 4716 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8007070 con 0x7fb6040703b0 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb601573640 4 set_mon_vals no callback set 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb5e37fe640 10 monclient: dump: 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.360 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _renew_subs 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6040be8b0 con 0x7fb6040703b0 2026-02-22T21:38:29.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 client.4716.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _renew_subs 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6040bee70 con 0x7fb6040703b0 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 1 librados: init done 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.345+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb604074da0 con 0x7fb6040703b0 2026-02-22T21:38:29.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.453+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb5f8007210 con 0x7fb6040703b0 2026-02-22T21:38:29.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.453+0000 7fb5e37fe640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 0x7fb5d007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:29.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.453+0000 7fb602575640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 0x7fb5d007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:29.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.453+0000 7fb602575640 10 monclient: get_auth_request con 0x7fb5d0078e50 auth_method 0 2026-02-22T21:38:29.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.454+0000 7fb602575640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 0x7fb5d007b310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec003fd0 tx=0x7fb5ec007dc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:29.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.454+0000 7fb5e37fe640 10 client.4716.objecter ms_handle_connect 0x7fb5d0078e50 2026-02-22T21:38:29.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb5f805d4d0 con 0x7fb6040703b0 2026-02-22T21:38:29.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 10 client.4716.objecter ms_dispatch2 0x7fb604000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 3 client.4716.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 3 client.4716.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 20 client.4716.objecter dump_active .. 0 homeless 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb5f8022020 con 0x7fb6040703b0 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:29.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.461+0000 7fb5e37fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:29.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.563+0000 7fb6090da640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:29.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.563+0000 7fb6090da640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:29.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.563+0000 7fb6090da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:29.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.563+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb60414e280 con 0x7fb6040703b0 2026-02-22T21:38:29.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.565+0000 7fb5e37fe640 1 -- 10.20.193.51:0/1809073413 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb5f8061490 con 0x7fb6040703b0 2026-02-22T21:38:29.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.565+0000 7fb5e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:29.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.565+0000 7fb5e37fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:29.567 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:29.567 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 10 librados: watch_flush enter 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 10 librados: watch_flush exit 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 20 client.4716.objecter shutdown clearing up homeless session... 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 10 client.4716.objecter successfully canceled tick 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 msgr2=0x7fb5d007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 0x7fb5d007b310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec003fd0 tx=0x7fb5ec007dc0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 10 monclient: shutdown 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 msgr2=0x7fb6041f46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800c700 tx=0x7fb5f800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 shutdown_connections 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb5d0078e50 0x7fb5d007b310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb60415c710 0x7fb6041e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb604150220 0x7fb6040bfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 --2- 10.20.193.51:0/1809073413 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb6040703b0 0x7fb6041f46f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 >> 10.20.193.51:0/1809073413 conn(0x7fb6040cd7b0 msgr2=0x7fb6040cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:29.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 shutdown_connections 2026-02-22T21:38:29.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 -- 10.20.193.51:0/1809073413 wait complete. 2026-02-22T21:38:29.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:29.566+0000 7fb6090da640 1 librados: shutdown 2026-02-22T21:38:32.578 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.643+0000 7fb908296640 10 monclient: get_monmap_and_config 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.643+0000 7fb908296640 10 monclient: build_initial_monmap 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.643+0000 7fb908296640 10 monclient: monmap: 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:32.644549+0000 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:32.644549+0000 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:32.645 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 Processor -- start 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 -- start start 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient: init 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient: _add_conns ranks=[0,5,3,2,4,1] 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb900150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb90015c710 0x7fb9000c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): picked mon.noname-c con 0x7fb90015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb900150600 0x7fb90006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): picked mon.noname-a con 0x7fb900150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90015a920 con 0x7fb900150220 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fb90015af60 con 0x7fb900150600 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fb90015b240 con 0x7fb90015c710 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.644+0000 7fb908296640 10 monclient(hunting): authenticate will time out at 1188.224850s 2026-02-22T21:38:32.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb900150600 0x7fb90006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90580a640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb90015c710 0x7fb9000c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb900150600 0x7fb90006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:45856/0 (socket says 10.20.193.51:45856) 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 1 -- 10.20.193.51:0/1090283561 learned_addr learned my addr 10.20.193.51:0/1090283561 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 10 monclient(hunting): get_auth_request con 0x7fb900150600 auth_method 0 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90680c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90580a640 10 monclient(hunting): get_auth_request con 0x7fb90015c710 auth_method 0 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90580a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90580a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.645+0000 7fb90580a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:32.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.646+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 10 monclient(hunting): handle_auth_done global_id 4719 payload 274 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 1 -- 10.20.193.51:0/1090283561 >> v2:10.20.193.125:3300/0 conn(0x7fb90015c710 msgr2=0x7fb9000c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 1 --2- 10.20.193.51:0/1090283561 >> v2:10.20.193.125:3300/0 conn(0x7fb90015c710 0x7fb9000c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 1 -- 10.20.193.51:0/1090283561 >> v1:10.20.193.51:6789/0 conn(0x7fb900150220 legacy=0x7fb90015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 10 monclient: _finish_hunting 0 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 1 monclient: found mon.noname-a 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.647+0000 7fb90680c640 1 -- 10.20.193.51:0/1090283561 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90015b520 con 0x7fb900150600 2026-02-22T21:38:32.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.649+0000 7fb90680c640 1 --2- 10.20.193.51:0/1090283561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150600 0x7fb90006fca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009e50 tx=0x7fb8fc02f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0884473a26e9c56 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 1 -- 10.20.193.51:0/1090283561 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc02fdf0 con 0x7fb900150600 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: dump: 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: _finish_auth 0 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: _check_auth_tickets 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 5 monclient: authenticate success, global_id 4719 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 1 -- 10.20.193.51:0/1090283561 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8fc0373f0 con 0x7fb900150600 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 1 -- 10.20.193.51:0/1090283561 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0377c0 con 0x7fb900150600 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 4 set_mon_vals no callback set 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb905009640 10 monclient: dump: 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:32.653 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 10 monclient: shutdown 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 -- 10.20.193.51:0/1090283561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150600 msgr2=0x7fb90006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 --2- 10.20.193.51:0/1090283561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150600 0x7fb90006fca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009e50 tx=0x7fb8fc02f330 comp rx=0 tx=0).stop 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 -- 10.20.193.51:0/1090283561 shutdown_connections 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 --2- 10.20.193.51:0/1090283561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150600 0x7fb90006fca0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 --2- 10.20.193.51:0/1090283561 >> v2:10.20.193.125:3300/0 conn(0x7fb90015c710 0x7fb9000c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 -- 10.20.193.51:0/1090283561 >> 10.20.193.51:0/1090283561 conn(0x7fb9000cd7b0 msgr2=0x7fb9000cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 -- 10.20.193.51:0/1090283561 shutdown_connections 2026-02-22T21:38:32.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.651+0000 7fb908296640 1 -- 10.20.193.51:0/1090283561 wait complete. 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient: build_initial_monmap 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient: monmap: 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 librados: starting msgr at 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 librados: starting objecter 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 Processor -- start 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 -- start start 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 librados: setting wanted keys 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 librados: calling monclient init 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient: init 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): picked mon.noname-a con 0x7fb900150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb900150600 0x7fb9001493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): picked mon.noname-c con 0x7fb900150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb90015c710 0x7fb90006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): picked mon.noname-b con 0x7fb90015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb9000701e0 con 0x7fb900150220 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb90015bae0 con 0x7fb90015c710 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb900070360 con 0x7fb900150600 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:32.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.652+0000 7fb908296640 10 monclient(hunting): authenticate will time out at 1188.232983s 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:45860/0 (socket says 10.20.193.51:45860) 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 1 -- 10.20.193.51:0/2274772157 learned_addr learned my addr 10.20.193.51:0/2274772157 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): get_auth_request con 0x7fb900150220 auth_method 0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90680c640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb90015c710 0x7fb90006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90580a640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb900150600 0x7fb9001493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90600b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90680c640 10 monclient(hunting): get_auth_request con 0x7fb90015c710 auth_method 0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90680c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90680c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90680c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90580a640 10 monclient(hunting): get_auth_request con 0x7fb900150600 auth_method 0 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90580a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90580a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.653+0000 7fb90580a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90680c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90580a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:32.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 10 monclient(hunting): handle_auth_done global_id 4725 payload 824 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 -- 10.20.193.51:0/2274772157 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb900150600 msgr2=0x7fb9001493f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb900150600 0x7fb9001493f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 -- 10.20.193.51:0/2274772157 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb90015c710 msgr2=0x7fb90006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb90015c710 0x7fb90006e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 10 monclient: _finish_hunting 0 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 monclient: found mon.noname-a 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.654+0000 7fb90600b640 1 -- 10.20.193.51:0/2274772157 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9001f4530 con 0x7fb900150220 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.655+0000 7fb90600b640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000c580 tx=0x7fb8f000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.655+0000 7fb8eeffd640 10 client.?.objecter ms_handle_connect 0x7fb900150220 2026-02-22T21:38:32.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.655+0000 7fb8eeffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:32.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0017070 con 0x7fb900150220 2026-02-22T21:38:32.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:32.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: dump: 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: _finish_auth 0 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: _check_auth_tickets 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 5 monclient: authenticate success, global_id 4725 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _renew_subs 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f00052d0 con 0x7fb900150220 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb900158730 con 0x7fb900150220 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f00036b0 con 0x7fb900150220 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb904808640 4 set_mon_vals no callback set 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb8eeffd640 10 monclient: dump: 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _renew_subs 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9001f7a10 con 0x7fb900150220 2026-02-22T21:38:32.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 1 librados: init done 2026-02-22T21:38:32.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:32.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:32.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.657+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb90006fca0 con 0x7fb900150220 2026-02-22T21:38:32.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.668+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb8f0003850 con 0x7fb900150220 2026-02-22T21:38:32.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.669+0000 7fb8eeffd640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 0x7fb8d007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:32.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.669+0000 7fb90580a640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 0x7fb8d007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:32.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.669+0000 7fb90580a640 10 monclient: get_auth_request con 0x7fb8d0078e50 auth_method 0 2026-02-22T21:38:32.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.669+0000 7fb8eeffd640 10 client.4725.objecter ms_handle_connect 0x7fb8d0078e50 2026-02-22T21:38:32.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.669+0000 7fb90580a640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 0x7fb8d007b310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9000c0ee0 tx=0x7fb8f4009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:32.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.670+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb8f005d1f0 con 0x7fb900150220 2026-02-22T21:38:32.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.670+0000 7fb8eeffd640 10 client.4725.objecter ms_dispatch2 0x7fb900000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:32.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.670+0000 7fb8eeffd640 3 client.4725.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:32.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.670+0000 7fb8eeffd640 3 client.4725.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:32.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.670+0000 7fb8eeffd640 20 client.4725.objecter dump_active .. 0 homeless 2026-02-22T21:38:32.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.773+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb8f0021080 con 0x7fb900150220 2026-02-22T21:38:32.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.773+0000 7fb8eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:32.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.773+0000 7fb8eeffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:32.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.873+0000 7fb908296640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:32.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.873+0000 7fb908296640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:32.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.873+0000 7fb908296640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:32.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.873+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb9001f39b0 con 0x7fb900150220 2026-02-22T21:38:32.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.875+0000 7fb8eeffd640 1 -- 10.20.193.51:0/2274772157 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb8f00611b0 con 0x7fb900150220 2026-02-22T21:38:32.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.875+0000 7fb8eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:32.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.875+0000 7fb8eeffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:32.877 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:32.877 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.876+0000 7fb908296640 10 librados: watch_flush enter 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.876+0000 7fb908296640 10 librados: watch_flush exit 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.876+0000 7fb908296640 20 client.4725.objecter shutdown clearing up homeless session... 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.876+0000 7fb908296640 10 client.4725.objecter successfully canceled tick 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 msgr2=0x7fb8d007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 0x7fb8d007b310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9000c0ee0 tx=0x7fb8f4009290 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 10 monclient: shutdown 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 msgr2=0x7fb9001f74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000c580 tx=0x7fb8f000ca50 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 shutdown_connections 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb8d0078e50 0x7fb8d007b310 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb90015c710 0x7fb90006e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb900150600 0x7fb9001493f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 --2- 10.20.193.51:0/2274772157 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb900150220 0x7fb9001f74d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 >> 10.20.193.51:0/2274772157 conn(0x7fb9000cd7b0 msgr2=0x7fb90015cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 shutdown_connections 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 -- 10.20.193.51:0/2274772157 wait complete. 2026-02-22T21:38:32.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:32.877+0000 7fb908296640 1 librados: shutdown 2026-02-22T21:38:35.888 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 10 monclient: get_monmap_and_config 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 10 monclient: build_initial_monmap 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 10 monclient: monmap: 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:35.956 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:35.956017+0000 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:35.956017+0000 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:35.957 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 1 Processor -- start 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 1 -- start start 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.955+0000 7fd11fd40640 10 monclient: init 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient: _add_conns ranks=[3,0,2,1,4,5] 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd118150220 0x7fd1180c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-a con 0x7fd118150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd118150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd11815c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:35.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd11815a920 con 0x7fd118150600 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd11815af60 con 0x7fd11815c710 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fd11815b240 con 0x7fd118150220 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11fd40640 10 monclient(hunting): authenticate will time out at 1191.536346s 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11dab5640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd118150220 0x7fd1180c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11e2b6640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fd11815c710 0x7fd11806fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:46970/0 (socket says 10.20.193.51:46970) 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.956+0000 7fd11e2b6640 1 -- 10.20.193.51:0/3743596495 learned_addr learned my addr 10.20.193.51:0/3743596495 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): get_auth_request con 0x7fd118150220 auth_method 0 2026-02-22T21:38:35.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.957+0000 7fd11cab3640 1 -- 10.20.193.51:0/3743596495 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973300893 0 0) 0x7fd11815af60 con 0x7fd11815c710 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.958+0000 7fd11cab3640 10 monclient(hunting): _negotiate 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.958+0000 7fd11cab3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.958+0000 7fd11cab3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.958+0000 7fd11cab3640 10 monclient(hunting): my global_id is 4664 2026-02-22T21:38:35.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.958+0000 7fd11cab3640 1 -- 10.20.193.51:0/3743596495 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd100003620 con 0x7fd11815c710 2026-02-22T21:38:35.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 10 monclient(hunting): handle_auth_done global_id 4728 payload 274 2026-02-22T21:38:35.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 1 -- 10.20.193.51:0/3743596495 >> v1:10.20.193.125:6789/0 conn(0x7fd11815c710 legacy=0x7fd11806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 1 -- 10.20.193.51:0/3743596495 >> v1:10.20.193.51:6789/0 conn(0x7fd118150600 legacy=0x7fd11815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 10 monclient: _finish_hunting 0 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 1 monclient: found mon.noname-a 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.959+0000 7fd11dab5640 1 -- 10.20.193.51:0/3743596495 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11815b520 con 0x7fd118150220 2026-02-22T21:38:35.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.960+0000 7fd11dab5640 1 --2- 10.20.193.51:0/3743596495 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd118150220 0x7fd1180c73a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1080098a0 tx=0x7fd10802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dd2f26031264c3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:35.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 1 -- 10.20.193.51:0/3743596495 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10803c070 con 0x7fd118150220 2026-02-22T21:38:35.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:35.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:35.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: dump: 2026-02-22T21:38:35.971 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: _finish_auth 0 2026-02-22T21:38:35.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: _check_auth_tickets 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 1 -- 10.20.193.51:0/3743596495 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10802fc10 con 0x7fd118150220 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11fd40640 5 monclient: authenticate success, global_id 4728 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11fd40640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 1 -- 10.20.193.51:0/3743596495 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108037440 con 0x7fd118150220 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11fd40640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11fd40640 4 set_mon_vals no callback set 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:35.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.962+0000 7fd11cab3640 10 monclient: dump: 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:35.974 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 10 monclient: shutdown 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 -- 10.20.193.51:0/3743596495 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd118150220 msgr2=0x7fd1180c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 --2- 10.20.193.51:0/3743596495 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd118150220 0x7fd1180c73a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1080098a0 tx=0x7fd10802ef00 comp rx=0 tx=0).stop 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 -- 10.20.193.51:0/3743596495 shutdown_connections 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 --2- 10.20.193.51:0/3743596495 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd118150220 0x7fd1180c73a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 -- 10.20.193.51:0/3743596495 >> 10.20.193.51:0/3743596495 conn(0x7fd1180cd7b0 msgr2=0x7fd1180cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 -- 10.20.193.51:0/3743596495 shutdown_connections 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 -- 10.20.193.51:0/3743596495 wait complete. 2026-02-22T21:38:35.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 10 monclient: build_initial_monmap 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 10 monclient: monmap: 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:35.976 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 librados: starting msgr at 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 librados: starting objecter 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.963+0000 7fd11fd40640 1 Processor -- start 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 -- start start 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 librados: setting wanted keys 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 librados: calling monclient init 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient: init 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:38:35.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd118150220 0x7fd11806e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-c con 0x7fd118150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd118150600 0x7fd1181e6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-b con 0x7fd118150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): picked mon.noname-a con 0x7fd11815c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd1180703b0 con 0x7fd11815c710 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd11815bae0 con 0x7fd118150600 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:35.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd118070530 con 0x7fd118150220 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11fd40640 10 monclient(hunting): authenticate will time out at 1191.544443s 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:45876/0 (socket says 10.20.193.51:45876) 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 1 -- 10.20.193.51:0/990019077 learned_addr learned my addr 10.20.193.51:0/990019077 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 10 monclient(hunting): get_auth_request con 0x7fd11815c710 auth_method 0 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11e2b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11dab5640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd118150220 0x7fd11806e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:35.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.964+0000 7fd11d2b4640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd118150600 0x7fd1181e6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11e2b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11e2b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11e2b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): get_auth_request con 0x7fd118150220 auth_method 0 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): get_auth_request con 0x7fd118150600 auth_method 0 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:35.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11dab5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.965+0000 7fd11d2b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 10 monclient(hunting): handle_auth_done global_id 4734 payload 824 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 -- 10.20.193.51:0/990019077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd118150220 msgr2=0x7fd11806e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd118150220 0x7fd11806e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 -- 10.20.193.51:0/990019077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd118150600 msgr2=0x7fd1181e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd118150600 0x7fd1181e6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:35.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 10 monclient: _finish_hunting 0 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 monclient: found mon.noname-a 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.966+0000 7fd11e2b6640 1 -- 10.20.193.51:0/990019077 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1180c75f0 con 0x7fd11815c710 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.967+0000 7fd11e2b6640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd11400cb40 tx=0x7fd11400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.967+0000 7fd1067fc640 10 client.?.objecter ms_handle_connect 0x7fd11815c710 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.967+0000 7fd1067fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11400ad50 con 0x7fd11815c710 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: dump: 2026-02-22T21:38:35.982 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: _finish_auth 0 2026-02-22T21:38:35.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: _check_auth_tickets 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd114012400 con 0x7fd11815c710 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 5 monclient: authenticate success, global_id 4734 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 10 monclient: _renew_subs 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1180c78d0 con 0x7fd11815c710 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1140127f0 con 0x7fd11815c710 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd107fff640 4 set_mon_vals no callback set 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd1067fc640 10 monclient: dump: 2026-02-22T21:38:35.984 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 10 monclient: _renew_subs 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:35.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1180c7e90 con 0x7fd11815c710 2026-02-22T21:38:35.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.968+0000 7fd11fd40640 1 librados: init done 2026-02-22T21:38:35.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.969+0000 7fd11fd40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:35.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.969+0000 7fd11fd40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:35.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.969+0000 7fd11fd40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:35.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.969+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1180c73a0 con 0x7fd11815c710 2026-02-22T21:38:35.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.980+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd114012990 con 0x7fd11815c710 2026-02-22T21:38:35.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.981+0000 7fd1067fc640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 0x7fd0e807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:35.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.981+0000 7fd11dab5640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 0x7fd0e807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:35.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.981+0000 7fd11dab5640 10 monclient: get_auth_request con 0x7fd0e8078e50 auth_method 0 2026-02-22T21:38:35.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.981+0000 7fd11dab5640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 0x7fd0e807b310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd108039000 tx=0x7fd108033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:35.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:35.981+0000 7fd1067fc640 10 client.4734.objecter ms_handle_connect 0x7fd0e8078e50 2026-02-22T21:38:36.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd11405e930 con 0x7fd11815c710 2026-02-22T21:38:36.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 10 client.4734.objecter ms_dispatch2 0x7fd118000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:36.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 3 client.4734.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:36.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 3 client.4734.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:36.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 20 client.4734.objecter dump_active .. 0 homeless 2026-02-22T21:38:36.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd1140628f0 con 0x7fd11815c710 2026-02-22T21:38:36.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:36.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.087+0000 7fd1067fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:36.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.187+0000 7fd11fd40640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:36.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.187+0000 7fd11fd40640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:36.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.187+0000 7fd11fd40640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:36.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.187+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd118074da0 con 0x7fd11815c710 2026-02-22T21:38:36.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.191+0000 7fd1067fc640 1 -- 10.20.193.51:0/990019077 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fd114066bf0 con 0x7fd11815c710 2026-02-22T21:38:36.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.191+0000 7fd1067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:36.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.191+0000 7fd1067fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:36.192 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:36.192 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 10 librados: watch_flush enter 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 10 librados: watch_flush exit 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 20 client.4734.objecter shutdown clearing up homeless session... 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 10 client.4734.objecter successfully canceled tick 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 msgr2=0x7fd0e807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 0x7fd0e807b310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd108039000 tx=0x7fd108033000 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 10 monclient: shutdown 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 msgr2=0x7fd1181e7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd11400cb40 tx=0x7fd11400a430 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 shutdown_connections 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0e8078e50 0x7fd0e807b310 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd11815c710 0x7fd1181e7160 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd118150600 0x7fd1181e6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 --2- 10.20.193.51:0/990019077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd118150220 0x7fd11806e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 >> 10.20.193.51:0/990019077 conn(0x7fd1180cd7b0 msgr2=0x7fd1180cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 shutdown_connections 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 -- 10.20.193.51:0/990019077 wait complete. 2026-02-22T21:38:36.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:36.192+0000 7fd11fd40640 1 librados: shutdown 2026-02-22T21:38:39.203 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:39.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 10 monclient: get_monmap_and_config 2026-02-22T21:38:39.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 10 monclient: build_initial_monmap 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 10 monclient: monmap: 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:39.269871+0000 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:39.269871+0000 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:39.271 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 1 Processor -- start 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 1 -- start start 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.269+0000 7f53baa2b640 10 monclient: init 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient: _add_conns ranks=[4,3,0,1,2,5] 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f53b4150220 0x7f53b40c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f53b4150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 0x7f53b415c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53b415c540 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b415c9b0 con 0x7f53b415c540 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f53b415a920 con 0x7f53b4150600 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:39.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f53b415af60 con 0x7f53b4150220 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53baa2b640 10 monclient(hunting): authenticate will time out at 1194.850164s 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 0x7f53b415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b3fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f53b4150220 0x7f53b40c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 0x7f53b415c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48974/0 (socket says 10.20.193.51:48974) 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 1 -- 10.20.193.51:0/2003726967 learned_addr learned my addr 10.20.193.51:0/2003726967 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 10 monclient(hunting): get_auth_request con 0x7f53b4150600 auth_method 0 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b3fff640 10 monclient(hunting): get_auth_request con 0x7f53b4150220 auth_method 0 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.270+0000 7f53b3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:39.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.271+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 10 monclient(hunting): handle_auth_done global_id 4702 payload 274 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 1 -- 10.20.193.51:0/2003726967 >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 msgr2=0x7f53b415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 1 --2- 10.20.193.51:0/2003726967 >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 0x7f53b415c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 1 -- 10.20.193.51:0/2003726967 >> v1:10.20.193.51:6789/0 conn(0x7f53b415c540 legacy=0x7f53b406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 1 monclient: found mon.noname-b 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:39.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:39.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.272+0000 7f53b3fff640 1 -- 10.20.193.51:0/2003726967 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b415b240 con 0x7f53b4150220 2026-02-22T21:38:39.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.273+0000 7f53b3fff640 1 --2- 10.20.193.51:0/2003726967 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150220 0x7f53b40c73a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f53b4070940 tx=0x7f53a002eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a578a66512087f1e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:39.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 1 -- 10.20.193.51:0/2003726967 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a003c070 con 0x7f53b4150220 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: dump: 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 1 -- 10.20.193.51:0/2003726967 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a002fb80 con 0x7f53b4150220 2026-02-22T21:38:39.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 5 monclient: authenticate success, global_id 4702 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 1 -- 10.20.193.51:0/2003726967 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0037440 con 0x7f53b4150220 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53b2ffd640 10 monclient: dump: 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 4 set_mon_vals no callback set 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 10 monclient: shutdown 2026-02-22T21:38:39.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 -- 10.20.193.51:0/2003726967 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150220 msgr2=0x7f53b40c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 --2- 10.20.193.51:0/2003726967 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150220 0x7f53b40c73a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f53b4070940 tx=0x7f53a002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 -- 10.20.193.51:0/2003726967 shutdown_connections 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 --2- 10.20.193.51:0/2003726967 >> v2:10.20.193.51:3300/0 conn(0x7f53b4150600 0x7f53b415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 --2- 10.20.193.51:0/2003726967 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150220 0x7f53b40c73a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 -- 10.20.193.51:0/2003726967 >> 10.20.193.51:0/2003726967 conn(0x7f53b40cd7b0 msgr2=0x7f53b40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.275+0000 7f53baa2b640 1 -- 10.20.193.51:0/2003726967 shutdown_connections 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 -- 10.20.193.51:0/2003726967 wait complete. 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 10 monclient: build_initial_monmap 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 10 monclient: monmap: 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:39.280 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 librados: starting msgr at 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 librados: starting objecter 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 Processor -- start 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 -- start start 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 librados: setting wanted keys 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 1 librados: calling monclient init 2026-02-22T21:38:39.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.276+0000 7f53baa2b640 10 monclient: init 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150600 0x7f53b41f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-b con 0x7f53b4150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415c540 0x7f53b414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): picked mon.noname-c con 0x7f53b415c540 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f53b415bae0 con 0x7f53b4150220 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f53b415b800 con 0x7f53b4150600 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40703b0 con 0x7f53b415c540 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53baa2b640 10 monclient(hunting): authenticate will time out at 1194.857091s 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150600 0x7f53b41f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48978/0 (socket says 10.20.193.51:48978) 2026-02-22T21:38:39.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 1 -- 10.20.193.51:0/4106927617 learned_addr learned my addr 10.20.193.51:0/4106927617 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): get_auth_request con 0x7f53b4150220 auth_method 0 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b8fa1640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415c540 0x7f53b414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): get_auth_request con 0x7f53b4150600 auth_method 0 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:39.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b8fa1640 10 monclient(hunting): get_auth_request con 0x7f53b415c540 auth_method 0 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b8fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b8fa1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b8fa1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.277+0000 7f53b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.278+0000 7f53b8fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.278+0000 7f53b8fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.278+0000 7f53b8fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 10 monclient(hunting): handle_auth_done global_id 4740 payload 824 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 -- 10.20.193.51:0/4106927617 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415c540 msgr2=0x7f53b414cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415c540 0x7f53b414cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 -- 10.20.193.51:0/4106927617 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150600 msgr2=0x7f53b41f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150600 0x7f53b41f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 10 monclient: _finish_hunting 0 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 monclient: found mon.noname-a 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.279+0000 7f53b3fff640 1 -- 10.20.193.51:0/4106927617 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b40bed40 con 0x7f53b4150220 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.280+0000 7f53b3fff640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53a003a040 tx=0x7f53a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.280+0000 7f53b0ff9640 10 client.?.objecter ms_handle_connect 0x7f53b4150220 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.280+0000 7f53b0ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a003c040 con 0x7f53b4150220 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:39.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: dump: 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: _finish_auth 0 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 5 monclient: authenticate success, global_id 4740 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a0042610 con 0x7f53b4150220 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00429e0 con 0x7f53b4150220 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b27fc640 4 set_mon_vals no callback set 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53b0ff9640 10 monclient: dump: 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:39.290 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 10 monclient: _renew_subs 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:39.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b40bf020 con 0x7f53b4150220 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 10 client.4740.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 10 monclient: _renew_subs 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b40bf5e0 con 0x7f53b4150220 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.281+0000 7f53baa2b640 1 librados: init done 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.282+0000 7f53baa2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.282+0000 7f53baa2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.282+0000 7f53baa2b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:39.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.282+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b415c000 con 0x7f53b4150220 2026-02-22T21:38:39.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.387+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f53a004b450 con 0x7f53b4150220 2026-02-22T21:38:39.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.387+0000 7f53b0ff9640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:39.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.388+0000 7f53b37fe640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:39.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.388+0000 7f53b37fe640 10 monclient: get_auth_request con 0x7f5384078e50 auth_method 0 2026-02-22T21:38:39.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.388+0000 7f53b0ff9640 10 client.4740.objecter ms_handle_connect 0x7f5384078e50 2026-02-22T21:38:39.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.388+0000 7f53b37fe640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f53b41e8b20 tx=0x7f53a4009340 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f53a002fb00 con 0x7f53b4150220 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 10 client.4740.objecter ms_dispatch2 0x7f53b4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 3 client.4740.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 3 client.4740.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 20 client.4740.objecter dump_active .. 0 homeless 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f53a00868a0 con 0x7f53b4150220 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:39.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.396+0000 7f53b0ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:39.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.496+0000 7f53baa2b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:39.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.496+0000 7f53baa2b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:39.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.496+0000 7f53baa2b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:39.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.496+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f53b4074da0 con 0x7f53b4150220 2026-02-22T21:38:39.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.498+0000 7f53b0ff9640 1 -- 10.20.193.51:0/4106927617 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f53a008aba0 con 0x7f53b4150220 2026-02-22T21:38:39.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.498+0000 7f53b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:39.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.498+0000 7f53b0ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:39.500 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:39.500 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 10 librados: watch_flush enter 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 10 librados: watch_flush exit 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 20 client.4740.objecter shutdown clearing up homeless session... 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 10 client.4740.objecter successfully canceled tick 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 msgr2=0x7f538407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f53b41e8b20 tx=0x7f53a4009340 comp rx=0 tx=0).stop 2026-02-22T21:38:39.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 10 monclient: shutdown 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 msgr2=0x7f53b41f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53a003a040 tx=0x7f53a00047c0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 shutdown_connections 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5384078e50 0x7f538407b310 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f53b415c540 0x7f53b414cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f53b4150600 0x7f53b41f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 --2- 10.20.193.51:0/4106927617 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f53b4150220 0x7f53b41f4f20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 >> 10.20.193.51:0/4106927617 conn(0x7f53b40cd7b0 msgr2=0x7f53b41590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 shutdown_connections 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 -- 10.20.193.51:0/4106927617 wait complete. 2026-02-22T21:38:39.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:39.499+0000 7f53baa2b640 1 librados: shutdown 2026-02-22T21:38:41.344 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:12.497 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.370 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:12.486 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.371 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:12.525 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.372 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:12.488 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.373 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:12.523 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.374 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:12.526 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.380 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:12.534 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.380 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:12.496 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:41.383 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:12.535 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:38:42.510 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.575+0000 7fa6591c3640 10 monclient: get_monmap_and_config 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.575+0000 7fa6591c3640 10 monclient: build_initial_monmap 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.575+0000 7fa6591c3640 10 monclient: monmap: 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:42.576592+0000 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:42.576592+0000 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:42.577 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 Processor -- start 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 -- start start 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient: init 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient: _add_conns ranks=[3,0,1,2,4,5] 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa6541527b0 0x7fa6540c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-a con 0x7fa6541527b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6540bf800 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fa6540703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65415ffa0 con 0x7fa6540bf800 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6540bea60 con 0x7fa6540703b0 2026-02-22T21:38:42.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa6540be810 con 0x7fa6541527b0 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.576+0000 7fa6591c3640 10 monclient(hunting): authenticate will time out at 1198.156943s 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa652d76640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa6541527b0 0x7fa6540c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa653577640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fa6540703b0 0x7fa65415f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:39232/0 (socket says 10.20.193.51:39232) 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa653577640 1 -- 10.20.193.51:0/2256677004 learned_addr learned my addr 10.20.193.51:0/2256677004 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa652d76640 10 monclient(hunting): get_auth_request con 0x7fa6541527b0 auth_method 0 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa652d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa652d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.577+0000 7fa652d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:42.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 1 -- 10.20.193.51:0/2256677004 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996788205 0 0) 0x7fa6540bea60 con 0x7fa6540703b0 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 10 monclient(hunting): _negotiate 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 10 monclient(hunting): my global_id is 4708 2026-02-22T21:38:42.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.578+0000 7fa651d74640 1 -- 10.20.193.51:0/2256677004 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa640003620 con 0x7fa6540703b0 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 10 monclient(hunting): handle_auth_done global_id 4743 payload 274 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 1 -- 10.20.193.51:0/2256677004 >> v1:10.20.193.68:6789/0 conn(0x7fa6540703b0 legacy=0x7fa65415f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 1 -- 10.20.193.51:0/2256677004 >> v1:10.20.193.51:6789/0 conn(0x7fa6540bf800 legacy=0x7fa65406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 10 monclient: _finish_hunting 0 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 1 monclient: found mon.noname-a 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.580+0000 7fa652d76640 1 -- 10.20.193.51:0/2256677004 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6540bed40 con 0x7fa6541527b0 2026-02-22T21:38:42.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.581+0000 7fa652d76640 1 --2- 10.20.193.51:0/2256677004 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa6540c9660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa64802ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24cbaae4db80606 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 1 -- 10.20.193.51:0/2256677004 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64803c070 con 0x7fa6541527b0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: dump: 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: _finish_auth 0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: _check_auth_tickets 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 1 -- 10.20.193.51:0/2256677004 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa64802fb70 con 0x7fa6541527b0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa6591c3640 5 monclient: authenticate success, global_id 4743 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa6591c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 1 -- 10.20.193.51:0/2256677004 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648037440 con 0x7fa6541527b0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:42.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa651d74640 10 monclient: dump: 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa6591c3640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.583+0000 7fa6591c3640 4 set_mon_vals no callback set 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 10 monclient: shutdown 2026-02-22T21:38:42.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- 10.20.193.51:0/2256677004 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 msgr2=0x7fa6540c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2256677004 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa6540c9660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa64802ee60 comp rx=0 tx=0).stop 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- 10.20.193.51:0/2256677004 shutdown_connections 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2256677004 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa6540c9660 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- 10.20.193.51:0/2256677004 >> 10.20.193.51:0/2256677004 conn(0x7fa6540cd7b0 msgr2=0x7fa6540cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- 10.20.193.51:0/2256677004 shutdown_connections 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- 10.20.193.51:0/2256677004 wait complete. 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 10 monclient: build_initial_monmap 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 10 monclient: monmap: 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:42.587 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 librados: starting msgr at 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 librados: starting objecter 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 Processor -- start 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 -- start start 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 librados: setting wanted keys 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 1 librados: calling monclient init 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.584+0000 7fa6591c3640 10 monclient: init 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa6540703b0 0x7fa65415ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-c con 0x7fa6540703b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa6540bf800 0x7fa65415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-b con 0x7fa6540bf800 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): picked mon.noname-a con 0x7fa6541527b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa6540bf300 con 0x7fa6541527b0 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa654152b90 con 0x7fa6540bf800 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa65406d710 con 0x7fa6540703b0 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa6591c3640 10 monclient(hunting): authenticate will time out at 1198.165270s 2026-02-22T21:38:42.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa6540bf800 0x7fa65415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48986/0 (socket says 10.20.193.51:48986) 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 1 -- 10.20.193.51:0/2928013337 learned_addr learned my addr 10.20.193.51:0/2928013337 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): get_auth_request con 0x7fa6541527b0 auth_method 0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652d76640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa6540703b0 0x7fa65415ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652575640 10 monclient(hunting): get_auth_request con 0x7fa6540bf800 auth_method 0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa652575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.585+0000 7fa653577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): get_auth_request con 0x7fa6540703b0 auth_method 0 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:42.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.586+0000 7fa652d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 -- 10.20.193.51:0/2928013337 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa6540703b0 msgr2=0x7fa65415ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa6540703b0 0x7fa65415ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 -- 10.20.193.51:0/2928013337 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa6540bf800 msgr2=0x7fa65415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa6540bf800 0x7fa65415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 10 monclient: _finish_hunting 0 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 monclient: found mon.noname-a 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.587+0000 7fa653577640 1 -- 10.20.193.51:0/2928013337 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65415ab30 con 0x7fa6541527b0 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.588+0000 7fa653577640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa64400caa0 tx=0x7fa64400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.588+0000 7fa6337fe640 10 client.?.objecter ms_handle_connect 0x7fa6541527b0 2026-02-22T21:38:42.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.588+0000 7fa6337fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64400ac40 con 0x7fa6541527b0 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 10 monclient: dump: 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:42.591 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.589+0000 7fa6337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 5 monclient: authenticate success, global_id 4749 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa64400ade0 con 0x7fa6541527b0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644012600 con 0x7fa6541527b0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6337fe640 10 monclient: dump: 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa651573640 4 set_mon_vals no callback set 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _renew_subs 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa654159cd0 con 0x7fa6541527b0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _renew_subs 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65415a290 con 0x7fa6541527b0 2026-02-22T21:38:42.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 1 librados: init done 2026-02-22T21:38:42.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:42.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:42.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.590+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6540bfc90 con 0x7fa6541527b0 2026-02-22T21:38:42.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.601+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa6440127a0 con 0x7fa6541527b0 2026-02-22T21:38:42.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.602+0000 7fa6337fe640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 0x7fa62007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:42.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.602+0000 7fa652d76640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 0x7fa62007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:42.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.602+0000 7fa652d76640 10 monclient: get_auth_request con 0x7fa620078e50 auth_method 0 2026-02-22T21:38:42.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.602+0000 7fa652d76640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 0x7fa62007b310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa648039000 tx=0x7fa648033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:42.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.602+0000 7fa6337fe640 10 client.4749.objecter ms_handle_connect 0x7fa620078e50 2026-02-22T21:38:42.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.604+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa64405e810 con 0x7fa6541527b0 2026-02-22T21:38:42.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.604+0000 7fa6337fe640 10 client.4749.objecter ms_dispatch2 0x7fa654000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:42.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.604+0000 7fa6337fe640 3 client.4749.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:42.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.604+0000 7fa6337fe640 3 client.4749.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:42.606 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.604+0000 7fa6337fe640 20 client.4749.objecter dump_active .. 0 homeless 2026-02-22T21:38:42.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.706+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa65415ab30 con 0x7fa6541527b0 2026-02-22T21:38:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.706+0000 7fa6337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:42.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.706+0000 7fa6337fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:42.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.806+0000 7fa6591c3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:42.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.806+0000 7fa6591c3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:42.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.806+0000 7fa6591c3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:42.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.806+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa65406e4c0 con 0x7fa6541527b0 2026-02-22T21:38:42.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.808+0000 7fa6337fe640 1 -- 10.20.193.51:0/2928013337 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa6440627d0 con 0x7fa6541527b0 2026-02-22T21:38:42.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:42.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6337fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:42.810 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:42.810 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:42.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 10 librados: watch_flush enter 2026-02-22T21:38:42.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 10 librados: watch_flush exit 2026-02-22T21:38:42.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 20 client.4749.objecter shutdown clearing up homeless session... 2026-02-22T21:38:42.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 10 client.4749.objecter successfully canceled tick 2026-02-22T21:38:42.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 msgr2=0x7fa62007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 0x7fa62007b310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa648039000 tx=0x7fa648033000 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 10 monclient: shutdown 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 msgr2=0x7fa65415d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa64400caa0 tx=0x7fa64400cf70 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 shutdown_connections 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa620078e50 0x7fa62007b310 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa6541527b0 0x7fa65415d9d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa6540bf800 0x7fa65415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 --2- 10.20.193.51:0/2928013337 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa6540703b0 0x7fa65415ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 >> 10.20.193.51:0/2928013337 conn(0x7fa6540cd7b0 msgr2=0x7fa65406ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 shutdown_connections 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 -- 10.20.193.51:0/2928013337 wait complete. 2026-02-22T21:38:42.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:42.809+0000 7fa6591c3640 1 librados: shutdown 2026-02-22T21:38:45.820 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.887+0000 7f420506d640 10 monclient: get_monmap_and_config 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.887+0000 7f420506d640 10 monclient: build_initial_monmap 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.888+0000 7f420506d640 10 monclient: monmap: 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:45.888672+0000 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:45.888672+0000 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:45.889 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:45.890 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:45.890 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.888+0000 7f420506d640 1 Processor -- start 2026-02-22T21:38:45.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.888+0000 7f420506d640 1 -- start start 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.888+0000 7f420506d640 10 monclient: init 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient: _add_conns ranks=[2,3,1,4,5,0] 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4200152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f420006d9d0 0x7f42000c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): picked mon.noname-a con 0x7f420006d9d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f42000c93d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420006f1d0 con 0x7f42000c93d0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420006de50 con 0x7f4200152ab0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f420006ed40 con 0x7f420006d9d0 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f420506d640 10 monclient(hunting): authenticate will time out at 1201.469086s 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fed76640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f4200152ab0 0x7f42000c5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:32964/0 (socket says 10.20.193.51:32964) 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fed76640 1 -- 10.20.193.51:0/2529422338 learned_addr learned my addr 10.20.193.51:0/2529422338 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41ff577640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f42000c93d0 0x7f4200158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:39248/0 (socket says 10.20.193.51:39248) 2026-02-22T21:38:45.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fe575640 1 --2- 10.20.193.51:0/2529422338 >> v2:10.20.193.51:3300/0 conn(0x7f420006d9d0 0x7f42000c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fe575640 10 monclient(hunting): get_auth_request con 0x7f420006d9d0 auth_method 0 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fe575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fe575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.889+0000 7f41fe575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703898616 0 0) 0x7f420006de50 con 0x7f4200152ab0 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 10 monclient(hunting): _negotiate 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 10 monclient(hunting): my global_id is 4676 2026-02-22T21:38:45.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.890+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41ec003620 con 0x7f4200152ab0 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157448049 0 0) 0x7f420006f1d0 con 0x7f42000c93d0 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 10 monclient(hunting): _negotiate 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 10 monclient(hunting): my global_id is 4714 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f420006de50 con 0x7f42000c93d0 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 10 monclient(hunting): handle_auth_done global_id 4752 payload 274 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 1 -- 10.20.193.51:0/2529422338 >> v1:10.20.193.125:6789/0 conn(0x7f4200152ab0 legacy=0x7f42000c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 1 -- 10.20.193.51:0/2529422338 >> v1:10.20.193.68:6789/0 conn(0x7f42000c93d0 legacy=0x7f4200158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 10 monclient: _finish_hunting 0 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 1 monclient: found mon.noname-a 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:45.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:45.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.891+0000 7f41fe575640 1 -- 10.20.193.51:0/2529422338 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420006e130 con 0x7f420006d9d0 2026-02-22T21:38:45.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.892+0000 7f41fe575640 1 --2- 10.20.193.51:0/2529422338 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42000c8e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098d0 tx=0x7f41e802ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c456bad122d026f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:45.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e803c070 con 0x7f420006d9d0 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 10 monclient: dump: 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 10 monclient: _finish_auth 0 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 10 monclient: _check_auth_tickets 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.893+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e802fbf0 con 0x7f420006d9d0 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 5 monclient: authenticate success, global_id 4752 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f41fdd74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f41fdd74640 1 -- 10.20.193.51:0/2529422338 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8037440 con 0x7f420006d9d0 2026-02-22T21:38:45.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f41fdd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f41fdd74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f41fdd74640 10 monclient: dump: 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 4 set_mon_vals no callback set 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 10 monclient: shutdown 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 -- 10.20.193.51:0/2529422338 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 msgr2=0x7f42000c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 --2- 10.20.193.51:0/2529422338 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42000c8e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098d0 tx=0x7f41e802ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 -- 10.20.193.51:0/2529422338 shutdown_connections 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 --2- 10.20.193.51:0/2529422338 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42000c8e90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 -- 10.20.193.51:0/2529422338 >> 10.20.193.51:0/2529422338 conn(0x7f42000cd7b0 msgr2=0x7f42000cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 -- 10.20.193.51:0/2529422338 shutdown_connections 2026-02-22T21:38:45.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 1 -- 10.20.193.51:0/2529422338 wait complete. 2026-02-22T21:38:45.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 10 monclient: build_initial_monmap 2026-02-22T21:38:45.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.894+0000 7f420506d640 10 monclient: monmap: 2026-02-22T21:38:45.899 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:45.899 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 librados: starting msgr at 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 librados: starting objecter 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 Processor -- start 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 -- start start 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 librados: setting wanted keys 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 librados: calling monclient init 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient: init 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): picked mon.noname-a con 0x7f420006d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f42000c93d0 0x7f42001f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): picked mon.noname-b con 0x7f42000c93d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4200152ab0 0x7f42000c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:45.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): picked mon.noname-c con 0x7f4200152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f420006e6f0 con 0x7f420006d9d0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f42000c9880 con 0x7f42000c93d0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f420006e9d0 con 0x7f4200152ab0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.895+0000 7f420506d640 10 monclient(hunting): authenticate will time out at 1201.475872s 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f42000c93d0 0x7f42001f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4200152ab0 0x7f42000c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48994/0 (socket says 10.20.193.51:48994) 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 1 -- 10.20.193.51:0/821556348 learned_addr learned my addr 10.20.193.51:0/821556348 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): get_auth_request con 0x7f420006d9d0 auth_method 0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): get_auth_request con 0x7f4200152ab0 auth_method 0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): get_auth_request con 0x7f42000c93d0 auth_method 0 2026-02-22T21:38:45.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41ff577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.896+0000 7f41fe575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 10 monclient(hunting): handle_auth_done global_id 4755 payload 824 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 -- 10.20.193.51:0/821556348 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4200152ab0 msgr2=0x7f42000c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4200152ab0 0x7f42000c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 -- 10.20.193.51:0/821556348 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f42000c93d0 msgr2=0x7f42001f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f42000c93d0 0x7f42001f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 10 monclient: _finish_hunting 0 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 monclient: found mon.noname-a 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:45.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.897+0000 7f41fed76640 1 -- 10.20.193.51:0/821556348 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42001f3c30 con 0x7f420006d9d0 2026-02-22T21:38:45.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.898+0000 7f41fed76640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c520 tx=0x7f41f400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:45.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.898+0000 7f41df7fe640 10 client.?.objecter ms_handle_connect 0x7f420006d9d0 2026-02-22T21:38:45.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.898+0000 7f41df7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4017070 con 0x7f420006d9d0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: dump: 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4005e00 con 0x7f420006d9d0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40072b0 con 0x7f420006d9d0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:45.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41fd573640 4 set_mon_vals no callback set 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f41df7fe640 10 monclient: dump: 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 5 monclient: authenticate success, global_id 4755 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 10 monclient: _renew_subs 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42001f3f10 con 0x7f420006d9d0 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 10 client.4755.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 10 monclient: _renew_subs 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42000c2ec0 con 0x7f420006d9d0 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.900+0000 7f420506d640 1 librados: init done 2026-02-22T21:38:45.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.901+0000 7f420506d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:45.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.901+0000 7f420506d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:45.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.901+0000 7f420506d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:45.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.901+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4200152e90 con 0x7f420006d9d0 2026-02-22T21:38:45.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f41f4007510 con 0x7f420006d9d0 2026-02-22T21:38:45.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41df7fe640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 0x7f41cc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:45.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41fe575640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 0x7f41cc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:45.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41fe575640 10 monclient: get_auth_request con 0x7f41cc078e00 auth_method 0 2026-02-22T21:38:45.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41df7fe640 10 client.4755.objecter ms_handle_connect 0x7f41cc078e00 2026-02-22T21:38:45.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.912+0000 7f41fe575640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 0x7f41cc07b2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41e8039000 tx=0x7f41e8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:45.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.914+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f41f405d4a0 con 0x7f420006d9d0 2026-02-22T21:38:45.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.914+0000 7f41df7fe640 10 client.4755.objecter ms_dispatch2 0x7f4200000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:45.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.914+0000 7f41df7fe640 3 client.4755.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:45.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.914+0000 7f41df7fe640 3 client.4755.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:45.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:45.914+0000 7f41df7fe640 20 client.4755.objecter dump_active .. 0 homeless 2026-02-22T21:38:46.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.013+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f41f4021020 con 0x7f420006d9d0 2026-02-22T21:38:46.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.013+0000 7f41df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:46.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.013+0000 7f41df7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:46.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.115+0000 7f420506d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:46.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.115+0000 7f420506d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:46.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.115+0000 7f420506d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:46.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.115+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f42000c6b00 con 0x7f420006d9d0 2026-02-22T21:38:46.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.118+0000 7f41df7fe640 1 -- 10.20.193.51:0/821556348 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f41f4061460 con 0x7f420006d9d0 2026-02-22T21:38:46.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.118+0000 7f41df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:46.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.118+0000 7f41df7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:46.119 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:46.119 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:46.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 10 librados: watch_flush enter 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 10 librados: watch_flush exit 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 20 client.4755.objecter shutdown clearing up homeless session... 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 10 client.4755.objecter successfully canceled tick 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 msgr2=0x7f41cc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 0x7f41cc07b2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41e8039000 tx=0x7f41e8033000 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 10 monclient: shutdown 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 msgr2=0x7f42001f3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c520 tx=0x7f41f400c9f0 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 shutdown_connections 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f41cc078e00 0x7f41cc07b2c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4200152ab0 0x7f42000c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f42000c93d0 0x7f42001f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 --2- 10.20.193.51:0/821556348 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f420006d9d0 0x7f42001f3560 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 >> 10.20.193.51:0/821556348 conn(0x7f42000cd7b0 msgr2=0x7f42000cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 shutdown_connections 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 -- 10.20.193.51:0/821556348 wait complete. 2026-02-22T21:38:46.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:46.119+0000 7f420506d640 1 librados: shutdown 2026-02-22T21:38:49.130 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:49.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 10 monclient: get_monmap_and_config 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 10 monclient: build_initial_monmap 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 10 monclient: monmap: 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:49.200173+0000 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:49.200173+0000 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 1 Processor -- start 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 1 -- start start 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.199+0000 7fe976d2c640 10 monclient: init 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient: _add_conns ranks=[4,2,1,0,3,5] 2026-02-22T21:38:49.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe970150220 0x7fe9700c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fe970150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe970150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe97015c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97015a920 con 0x7fe97015c710 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97015af60 con 0x7fe970150600 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe97015b240 con 0x7fe970150220 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe976d2c640 10 monclient(hunting): authenticate will time out at 1204.780504s 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.200+0000 7fe974aa1640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe970150220 0x7fe9700c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe967fff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fe970150600 0x7fe97015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:34242/0 (socket says 10.20.193.51:34242) 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe967fff640 1 -- 10.20.193.51:0/620309983 learned_addr learned my addr 10.20.193.51:0/620309983 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): get_auth_request con 0x7fe970150220 auth_method 0 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:49.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.201+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 1 -- 10.20.193.51:0/620309983 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575383756 0 0) 0x7fe97015af60 con 0x7fe970150600 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 10 monclient(hunting): _negotiate 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 10 monclient(hunting): my global_id is 4682 2026-02-22T21:38:49.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe9677fe640 1 -- 10.20.193.51:0/620309983 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe95c003620 con 0x7fe970150600 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 10 monclient(hunting): handle_auth_done global_id 4720 payload 274 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 1 -- 10.20.193.51:0/620309983 >> v1:10.20.193.125:6789/0 conn(0x7fe970150600 legacy=0x7fe97015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 1 -- 10.20.193.51:0/620309983 >> v1:10.20.193.68:6789/0 conn(0x7fe97015c710 legacy=0x7fe97006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 10 monclient: _finish_hunting 0 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 1 monclient: found mon.noname-b 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.202+0000 7fe974aa1640 1 -- 10.20.193.51:0/620309983 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97015b520 con 0x7fe970150220 2026-02-22T21:38:49.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.204+0000 7fe974aa1640 1 --2- 10.20.193.51:0/620309983 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe970150220 0x7fe9700c73a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9600098a0 tx=0x7fe96002ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ff59bc5ebdc74a server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 1 -- 10.20.193.51:0/620309983 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96003c070 con 0x7fe970150220 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: dump: 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.207 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 1 -- 10.20.193.51:0/620309983 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96002fc10 con 0x7fe970150220 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 1 -- 10.20.193.51:0/620309983 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960037440 con 0x7fe970150220 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 5 monclient: authenticate success, global_id 4720 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 4 set_mon_vals no callback set 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe9677fe640 10 monclient: dump: 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:49.208 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 10 monclient: shutdown 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 -- 10.20.193.51:0/620309983 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe970150220 msgr2=0x7fe9700c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 --2- 10.20.193.51:0/620309983 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe970150220 0x7fe9700c73a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9600098a0 tx=0x7fe96002ef00 comp rx=0 tx=0).stop 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 -- 10.20.193.51:0/620309983 shutdown_connections 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 --2- 10.20.193.51:0/620309983 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe970150220 0x7fe9700c73a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 -- 10.20.193.51:0/620309983 >> 10.20.193.51:0/620309983 conn(0x7fe9700cd7b0 msgr2=0x7fe9700cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 -- 10.20.193.51:0/620309983 shutdown_connections 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.206+0000 7fe976d2c640 1 -- 10.20.193.51:0/620309983 wait complete. 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 10 monclient: build_initial_monmap 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 10 monclient: monmap: 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 librados: starting msgr at 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 librados: starting objecter 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 Processor -- start 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 -- start start 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 librados: setting wanted keys 2026-02-22T21:38:49.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 1 librados: calling monclient init 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.207+0000 7fe976d2c640 10 monclient: init 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe970150220 0x7fe9701f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-a con 0x7fe970150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe970150600 0x7fe97006e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fe970150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fe97015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe9700703b0 con 0x7fe970150220 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe97015bae0 con 0x7fe97015c710 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:49.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe970070530 con 0x7fe970150600 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe976d2c640 10 monclient(hunting): authenticate will time out at 1204.788314s 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe967fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe970150600 0x7fe97006e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe974aa1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe970150220 0x7fe9701f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36550/0 (socket says 10.20.193.51:36550) 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 1 -- 10.20.193.51:0/4267127481 learned_addr learned my addr 10.20.193.51:0/4267127481 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 10 monclient(hunting): get_auth_request con 0x7fe97015c710 auth_method 0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe9752a2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe967fff640 10 monclient(hunting): get_auth_request con 0x7fe970150600 auth_method 0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe967fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe967fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe974aa1640 10 monclient(hunting): get_auth_request con 0x7fe970150220 auth_method 0 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe974aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe974aa1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.208+0000 7fe974aa1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe9752a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe9752a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe9752a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:49.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.209+0000 7fe974aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 10 monclient(hunting): handle_auth_done global_id 4726 payload 824 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 -- 10.20.193.51:0/4267127481 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe970150600 msgr2=0x7fe97006e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe970150600 0x7fe97006e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 -- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe970150220 msgr2=0x7fe9701f4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe970150220 0x7fe9701f4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 10 monclient: _finish_hunting 0 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 monclient: found mon.noname-b 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.210+0000 7fe9752a2640 1 -- 10.20.193.51:0/4267127481 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9700c7a00 con 0x7fe97015c710 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.211+0000 7fe9752a2640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00ccb0 tx=0x7fe96c00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.211+0000 7fe9657fa640 10 client.?.objecter ms_handle_connect 0x7fe97015c710 2026-02-22T21:38:49.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.211+0000 7fe9657fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c003a70 con 0x7fe97015c710 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 10 monclient: dump: 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 10 monclient: _finish_auth 0 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 10 monclient: _check_auth_tickets 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe9657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 5 monclient: authenticate success, global_id 4726 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 10 monclient: _renew_subs 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9700c7ce0 con 0x7fe97015c710 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 10 monclient: _renew_subs 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9700c82a0 con 0x7fe97015c710 2026-02-22T21:38:49.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.212+0000 7fe976d2c640 1 librados: init done 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c003c10 con 0x7fe97015c710 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0050d0 con 0x7fe97015c710 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe966ffd640 4 set_mon_vals no callback set 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe9657fa640 10 monclient: dump: 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe976d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe976d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe976d2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.213+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe970074da0 con 0x7fe97015c710 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.224+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe96c005270 con 0x7fe97015c710 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.225+0000 7fe9657fa640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 0x7fe94007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.225+0000 7fe974aa1640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 0x7fe94007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.225+0000 7fe974aa1640 10 monclient: get_auth_request con 0x7fe940078e00 auth_method 0 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.225+0000 7fe974aa1640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 0x7fe94007b2c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe960039000 tx=0x7fe960033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:49.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.225+0000 7fe9657fa640 10 client.4726.objecter ms_handle_connect 0x7fe940078e00 2026-02-22T21:38:49.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.226+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe96c05e7a0 con 0x7fe97015c710 2026-02-22T21:38:49.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.226+0000 7fe9657fa640 10 client.4726.objecter ms_dispatch2 0x7fe970000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:49.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.226+0000 7fe9657fa640 3 client.4726.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:49.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.226+0000 7fe9657fa640 3 client.4726.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:49.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.226+0000 7fe9657fa640 20 client.4726.objecter dump_active .. 0 homeless 2026-02-22T21:38:49.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.325+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe96c062760 con 0x7fe97015c710 2026-02-22T21:38:49.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.325+0000 7fe9657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:49.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.325+0000 7fe9657fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:49.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.424+0000 7fe976d2c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:49.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.425+0000 7fe976d2c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:49.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.425+0000 7fe976d2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:49.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.425+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe9701593f0 con 0x7fe97015c710 2026-02-22T21:38:49.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.427+0000 7fe9657fa640 1 -- 10.20.193.51:0/4267127481 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe96c066a60 con 0x7fe97015c710 2026-02-22T21:38:49.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.427+0000 7fe9657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:49.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.427+0000 7fe9657fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:49.429 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:49.429 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 10 librados: watch_flush enter 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 10 librados: watch_flush exit 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 20 client.4726.objecter shutdown clearing up homeless session... 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 10 client.4726.objecter successfully canceled tick 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 msgr2=0x7fe94007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 0x7fe94007b2c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe960039000 tx=0x7fe960033000 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 10 monclient: shutdown 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 msgr2=0x7fe9700c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00ccb0 tx=0x7fe96c00a430 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 shutdown_connections 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe940078e00 0x7fe94007b2c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe97015c710 0x7fe9700c8aa0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe970150600 0x7fe97006e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 --2- 10.20.193.51:0/4267127481 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe970150220 0x7fe9701f4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 >> 10.20.193.51:0/4267127481 conn(0x7fe9700cd7b0 msgr2=0x7fe9700cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 shutdown_connections 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 -- 10.20.193.51:0/4267127481 wait complete. 2026-02-22T21:38:49.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:49.428+0000 7fe976d2c640 1 librados: shutdown 2026-02-22T21:38:52.438 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:52.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 10 monclient: get_monmap_and_config 2026-02-22T21:38:52.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 10 monclient: build_initial_monmap 2026-02-22T21:38:52.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 10 monclient: monmap: 2026-02-22T21:38:52.507 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:52.507 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:52.506732+0000 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:52.506732+0000 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:52.508 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 1 Processor -- start 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 1 -- start start 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.506+0000 7fb63ecda640 10 monclient: init 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient: _add_conns ranks=[2,5,4,3,0,1] 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb638150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb63815c710 0x7fb6380c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-c con 0x7fb63815c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:38:52.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb638150600 0x7fb63806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-b con 0x7fb638150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb63815a920 con 0x7fb638150220 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fb63815af60 con 0x7fb638150600 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fb63815b240 con 0x7fb63815c710 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63ecda640 10 monclient(hunting): authenticate will time out at 1208.087133s 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb62ffff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fb63815c710 0x7fb6380c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb638150600 0x7fb63806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb638150600 0x7fb63806fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36552/0 (socket says 10.20.193.51:36552) 2026-02-22T21:38:52.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 1 -- 10.20.193.51:0/537855667 learned_addr learned my addr 10.20.193.51:0/537855667 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:52.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 10 monclient(hunting): get_auth_request con 0x7fb638150600 auth_method 0 2026-02-22T21:38:52.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:52.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:52.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.507+0000 7fb63d250640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:52.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): get_auth_request con 0x7fb63815c710 auth_method 0 2026-02-22T21:38:52.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:52.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.508+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 10 monclient(hunting): handle_auth_done global_id 4729 payload 274 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 1 -- 10.20.193.51:0/537855667 >> v2:10.20.193.125:3300/0 conn(0x7fb63815c710 msgr2=0x7fb6380c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 1 --2- 10.20.193.51:0/537855667 >> v2:10.20.193.125:3300/0 conn(0x7fb63815c710 0x7fb6380c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 1 -- 10.20.193.51:0/537855667 >> v1:10.20.193.125:6789/0 conn(0x7fb638150220 legacy=0x7fb63815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 10 monclient: _finish_hunting 0 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 1 monclient: found mon.noname-b 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.509+0000 7fb63d250640 1 -- 10.20.193.51:0/537855667 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63815b520 con 0x7fb638150600 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.510+0000 7fb63d250640 1 --2- 10.20.193.51:0/537855667 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb638150600 0x7fb63806fca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb634009e50 tx=0x7fb63402f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d833b3e5f40420be server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 1 -- 10.20.193.51:0/537855667 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63402fdf0 con 0x7fb638150600 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: dump: 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb63ecda640 5 monclient: authenticate success, global_id 4729 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb63ecda640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 1 -- 10.20.193.51:0/537855667 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6340373f0 con 0x7fb638150600 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 1 -- 10.20.193.51:0/537855667 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6340377c0 con 0x7fb638150600 2026-02-22T21:38:52.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb62f7fe640 10 monclient: dump: 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb63ecda640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.511+0000 7fb63ecda640 4 set_mon_vals no callback set 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 10 monclient: shutdown 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- 10.20.193.51:0/537855667 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb638150600 msgr2=0x7fb63806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 --2- 10.20.193.51:0/537855667 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb638150600 0x7fb63806fca0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb634009e50 tx=0x7fb63402f330 comp rx=0 tx=0).stop 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- 10.20.193.51:0/537855667 shutdown_connections 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 --2- 10.20.193.51:0/537855667 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb638150600 0x7fb63806fca0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 --2- 10.20.193.51:0/537855667 >> v2:10.20.193.125:3300/0 conn(0x7fb63815c710 0x7fb6380c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- 10.20.193.51:0/537855667 >> 10.20.193.51:0/537855667 conn(0x7fb6380cd7b0 msgr2=0x7fb6380cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- 10.20.193.51:0/537855667 shutdown_connections 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- 10.20.193.51:0/537855667 wait complete. 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 10 monclient: build_initial_monmap 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 10 monclient: monmap: 2026-02-22T21:38:52.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 librados: starting msgr at 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 librados: starting objecter 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 Processor -- start 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 -- start start 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 librados: setting wanted keys 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 1 librados: calling monclient init 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.512+0000 7fb63ecda640 10 monclient: init 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb638150220 0x7fb638159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-a con 0x7fb638150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb638150600 0x7fb6381e9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-c con 0x7fb638150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): picked mon.noname-b con 0x7fb63815c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb6380701e0 con 0x7fb638150220 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb63815bae0 con 0x7fb63815c710 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:52.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb638070360 con 0x7fb638150600 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ecda640 10 monclient(hunting): authenticate will time out at 1208.093356s 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ca4f640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb638150220 0x7fb638159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb62ffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb638150600 0x7fb6381e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:36556/0 (socket says 10.20.193.51:36556) 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 1 -- 10.20.193.51:0/2199726711 learned_addr learned my addr 10.20.193.51:0/2199726711 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 10 monclient(hunting): get_auth_request con 0x7fb63815c710 auth_method 0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63d250640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb62ffff640 10 monclient(hunting): get_auth_request con 0x7fb638150600 auth_method 0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb62ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb62ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb62ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ca4f640 10 monclient(hunting): get_auth_request con 0x7fb638150220 auth_method 0 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ca4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ca4f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.513+0000 7fb63ca4f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63d250640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63ca4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63ca4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb63ca4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.514+0000 7fb62ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 -- 10.20.193.51:0/2199726711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb638150600 msgr2=0x7fb6381e9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb638150600 0x7fb6381e9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 -- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb638150220 msgr2=0x7fb638159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb638150220 0x7fb638159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 10 monclient: _finish_hunting 0 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 monclient: found mon.noname-b 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.515+0000 7fb63d250640 1 -- 10.20.193.51:0/2199726711 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb638157840 con 0x7fb63815c710 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.516+0000 7fb63d250640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb63402f860 tx=0x7fb634004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.516+0000 7fb62d7fa640 10 client.?.objecter ms_handle_connect 0x7fb63815c710 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.516+0000 7fb62d7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634046020 con 0x7fb63815c710 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: dump: 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: _finish_auth 0 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:38:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 5 monclient: authenticate success, global_id 4732 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 10 monclient: _renew_subs 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb638157b20 con 0x7fb63815c710 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 10 monclient: _renew_subs 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6381580e0 con 0x7fb63815c710 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb63ecda640 1 librados: init done 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6340373f0 con 0x7fb63815c710 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634009510 con 0x7fb63815c710 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62effd640 4 set_mon_vals no callback set 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.517+0000 7fb62d7fa640 10 monclient: dump: 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.518+0000 7fb63ecda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.518+0000 7fb63ecda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.518+0000 7fb63ecda640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.518+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63806fca0 con 0x7fb63815c710 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.529+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb634007bb0 con 0x7fb63815c710 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.529+0000 7fb62d7fa640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 0x7fb60807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.529+0000 7fb63ca4f640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 0x7fb60807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.529+0000 7fb63ca4f640 10 monclient: get_auth_request con 0x7fb608078e00 auth_method 0 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.530+0000 7fb62d7fa640 10 client.4732.objecter ms_handle_connect 0x7fb608078e00 2026-02-22T21:38:52.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.530+0000 7fb63ca4f640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 0x7fb60807b2c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6280051f0 tx=0x7fb628007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:52.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.531+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb6340bd4f0 con 0x7fb63815c710 2026-02-22T21:38:52.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.531+0000 7fb62d7fa640 10 client.4732.objecter ms_dispatch2 0x7fb638000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:52.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.531+0000 7fb62d7fa640 3 client.4732.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:52.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.531+0000 7fb62d7fa640 3 client.4732.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:52.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.531+0000 7fb62d7fa640 20 client.4732.objecter dump_active .. 0 homeless 2026-02-22T21:38:52.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.631+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb634050030 con 0x7fb63815c710 2026-02-22T21:38:52.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.632+0000 7fb62d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:52.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.632+0000 7fb62d7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:52.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.732+0000 7fb63ecda640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:52.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.732+0000 7fb63ecda640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:52.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.732+0000 7fb63ecda640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:52.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.732+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb638074da0 con 0x7fb63815c710 2026-02-22T21:38:52.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb62d7fa640 1 -- 10.20.193.51:0/2199726711 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb634087d80 con 0x7fb63815c710 2026-02-22T21:38:52.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb62d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:52.736 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb62d7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:52.736 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:52.736 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 10 librados: watch_flush enter 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 10 librados: watch_flush exit 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 20 client.4732.objecter shutdown clearing up homeless session... 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 10 client.4732.objecter successfully canceled tick 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 msgr2=0x7fb60807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 0x7fb60807b2c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6280051f0 tx=0x7fb628007f30 comp rx=0 tx=0).stop 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 10 monclient: shutdown 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 msgr2=0x7fb6381e9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.735+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb63402f860 tx=0x7fb634004b40 comp rx=0 tx=0).stop 2026-02-22T21:38:52.737 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 shutdown_connections 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb608078e00 0x7fb60807b2c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb63815c710 0x7fb6381e9f60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb638150600 0x7fb6381e9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 --2- 10.20.193.51:0/2199726711 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb638150220 0x7fb638159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 >> 10.20.193.51:0/2199726711 conn(0x7fb6380cd7b0 msgr2=0x7fb63815cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 shutdown_connections 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 -- 10.20.193.51:0/2199726711 wait complete. 2026-02-22T21:38:52.738 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:52.736+0000 7fb63ecda640 1 librados: shutdown 2026-02-22T21:38:55.746 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:55.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 10 monclient: get_monmap_and_config 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 10 monclient: build_initial_monmap 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 10 monclient: monmap: 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:55.814113+0000 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:55.814113+0000 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 1 Processor -- start 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 1 -- start start 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.813+0000 7f5e18e57640 10 monclient: init 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient: _add_conns ranks=[1,3,2,5,0,4] 2026-02-22T21:38:55.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f5e14150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5e1415c710 0x7f5e140c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-a con 0x7f5e1415c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e14150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1415a920 con 0x7f5e14150220 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1415af60 con 0x7f5e14150600 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f5e1415b240 con 0x7f5e1415c710 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e18e57640 10 monclient(hunting): authenticate will time out at 1211.394531s 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.814+0000 7f5e11d74640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5e1415c710 0x7f5e140c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e12575640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f5e14150220 0x7f5e1415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:38508/0 (socket says 10.20.193.51:38508) 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e12575640 1 -- 10.20.193.51:0/497410904 learned_addr learned my addr 10.20.193.51:0/497410904 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): get_auth_request con 0x7f5e1415c710 auth_method 0 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:55.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.815+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583758258 0 0) 0x7f5e1415a920 con 0x7f5e14150220 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _negotiate 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): my global_id is 4735 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e00003620 con 0x7f5e14150220 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424887911 0 0) 0x7f5e1415af60 con 0x7f5e14150600 2026-02-22T21:38:55.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _negotiate 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 10 monclient(hunting): my global_id is 4697 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e1415a920 con 0x7f5e14150600 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 10 monclient(hunting): handle_auth_done global_id 4764 payload 274 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 1 -- 10.20.193.51:0/497410904 >> v1:10.20.193.125:6789/0 conn(0x7f5e14150600 legacy=0x7f5e1406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 1 -- 10.20.193.51:0/497410904 >> v1:10.20.193.68:6789/0 conn(0x7f5e14150220 legacy=0x7f5e1415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 10 monclient: _finish_hunting 0 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 1 monclient: found mon.noname-a 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.816+0000 7f5e11d74640 1 -- 10.20.193.51:0/497410904 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1415b520 con 0x7f5e1415c710 2026-02-22T21:38:55.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.817+0000 7f5e11d74640 1 --2- 10.20.193.51:0/497410904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e1415c710 0x7f5e140c73a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098d0 tx=0x7f5dfc02ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96913927d63eb27 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.818+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc03c070 con 0x7f5e1415c710 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.818+0000 7f5e11573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.818+0000 7f5e11573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.818+0000 7f5e11573640 10 monclient: dump: 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: _finish_auth 0 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: _check_auth_tickets 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dfc02fbf0 con 0x7f5e1415c710 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 1 -- 10.20.193.51:0/497410904 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc037440 con 0x7f5e1415c710 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 5 monclient: authenticate success, global_id 4764 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 4 set_mon_vals no callback set 2026-02-22T21:38:55.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e11573640 10 monclient: dump: 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:55.821 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 10 monclient: shutdown 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 -- 10.20.193.51:0/497410904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e1415c710 msgr2=0x7f5e140c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 --2- 10.20.193.51:0/497410904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e1415c710 0x7f5e140c73a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098d0 tx=0x7f5dfc02ef20 comp rx=0 tx=0).stop 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 -- 10.20.193.51:0/497410904 shutdown_connections 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 --2- 10.20.193.51:0/497410904 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e1415c710 0x7f5e140c73a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 -- 10.20.193.51:0/497410904 >> 10.20.193.51:0/497410904 conn(0x7f5e140cd7b0 msgr2=0x7f5e140cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 -- 10.20.193.51:0/497410904 shutdown_connections 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 -- 10.20.193.51:0/497410904 wait complete. 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 10 monclient: build_initial_monmap 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 10 monclient: monmap: 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 librados: starting msgr at 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.819+0000 7f5e18e57640 1 librados: starting objecter 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 Processor -- start 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 -- start start 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 librados: setting wanted keys 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 librados: calling monclient init 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient: init 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:55.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5e14150220 0x7f5e140cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-c con 0x7f5e14150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-a con 0x7f5e14150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5e1415c710 0x7f5e141f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1415c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f5e140703b0 con 0x7f5e14150600 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1415bae0 con 0x7f5e1415c710 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5e14070530 con 0x7f5e14150220 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e18e57640 10 monclient(hunting): authenticate will time out at 1211.400496s 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:40460/0 (socket says 10.20.193.51:40460) 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 1 -- 10.20.193.51:0/2399244409 learned_addr learned my addr 10.20.193.51:0/2399244409 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 10 monclient(hunting): get_auth_request con 0x7f5e14150600 auth_method 0 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e11d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.820+0000 7f5e12d76640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5e1415c710 0x7f5e141f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:55.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5e14150220 0x7f5e140cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e11d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): get_auth_request con 0x7f5e1415c710 auth_method 0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): get_auth_request con 0x7f5e14150220 auth_method 0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.821+0000 7f5e12575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 -- 10.20.193.51:0/2399244409 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5e14150220 msgr2=0x7f5e140cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5e14150220 0x7f5e140cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 -- 10.20.193.51:0/2399244409 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5e1415c710 msgr2=0x7f5e141f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5e1415c710 0x7f5e141f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 10 monclient: _finish_hunting 0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 monclient: found mon.noname-a 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.822+0000 7f5e11d74640 1 -- 10.20.193.51:0/2399244409 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e141e6db0 con 0x7f5e14150600 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.823+0000 7f5e11d74640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098d0 tx=0x7f5dfc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:55.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.823+0000 7f5dfaffd640 10 client.?.objecter ms_handle_connect 0x7f5e14150600 2026-02-22T21:38:55.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.823+0000 7f5dfaffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc046020 con 0x7f5e14150600 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: dump: 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: _finish_auth 0 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: _check_auth_tickets 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dfc042650 con 0x7f5e14150600 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc03c040 con 0x7f5e14150600 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e10d72640 4 set_mon_vals no callback set 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5dfaffd640 10 monclient: dump: 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:55.826 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 5 monclient: authenticate success, global_id 4767 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 10 monclient: _renew_subs 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e141e7090 con 0x7f5e14150600 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 10 client.4767.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 10 monclient: _renew_subs 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e141e7650 con 0x7f5e14150600 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.824+0000 7f5e18e57640 1 librados: init done 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.825+0000 7f5e18e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.825+0000 7f5e18e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.825+0000 7f5e18e57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:55.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.825+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e140c73a0 con 0x7f5e14150600 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.932+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f5dfc04a3b0 con 0x7f5e14150600 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.933+0000 7f5dfaffd640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 0x7f5de007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.933+0000 7f5e12575640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 0x7f5de007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.933+0000 7f5e12575640 10 monclient: get_auth_request con 0x7f5de0078e50 auth_method 0 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.933+0000 7f5e12575640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 0x7f5de007b310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5e08007b60 tx=0x7f5e08007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:55.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.933+0000 7f5dfaffd640 10 client.4767.objecter ms_handle_connect 0x7f5de0078e50 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.941+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f5dfc02fab0 con 0x7f5e14150600 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.941+0000 7f5dfaffd640 10 client.4767.objecter ms_dispatch2 0x7f5e14000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.941+0000 7f5dfaffd640 3 client.4767.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.941+0000 7f5dfaffd640 3 client.4767.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.942+0000 7f5dfaffd640 20 client.4767.objecter dump_active .. 0 homeless 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.942+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f5dfc050020 con 0x7f5e14150600 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.942+0000 7f5dfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:55.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:55.942+0000 7f5dfaffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:56.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.042+0000 7f5e18e57640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:56.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.042+0000 7f5e18e57640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:56.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.042+0000 7f5e18e57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:38:56.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.042+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5e140c4270 con 0x7f5e14150600 2026-02-22T21:38:56.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.044+0000 7f5dfaffd640 1 -- 10.20.193.51:0/2399244409 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f5dfc086970 con 0x7f5e14150600 2026-02-22T21:38:56.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.044+0000 7f5dfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:56.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.044+0000 7f5dfaffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:56.046 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:56.046 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:56.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 10 librados: watch_flush enter 2026-02-22T21:38:56.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 10 librados: watch_flush exit 2026-02-22T21:38:56.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 20 client.4767.objecter shutdown clearing up homeless session... 2026-02-22T21:38:56.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 10 client.4767.objecter successfully canceled tick 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 msgr2=0x7f5de007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 0x7f5de007b310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5e08007b60 tx=0x7f5e08007ab0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 10 monclient: shutdown 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 msgr2=0x7f5e1406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098d0 tx=0x7f5dfc0047c0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 shutdown_connections 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5de0078e50 0x7f5de007b310 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5e1415c710 0x7f5e141f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5e14150600 0x7f5e1406e0e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 --2- 10.20.193.51:0/2399244409 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5e14150220 0x7f5e140cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 >> 10.20.193.51:0/2399244409 conn(0x7f5e140cd7b0 msgr2=0x7f5e140cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 shutdown_connections 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 -- 10.20.193.51:0/2399244409 wait complete. 2026-02-22T21:38:56.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:56.046+0000 7f5e18e57640 1 librados: shutdown 2026-02-22T21:38:59.057 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.121+0000 7f72647a9640 10 monclient: get_monmap_and_config 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.121+0000 7f72647a9640 10 monclient: build_initial_monmap 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.121+0000 7f72647a9640 10 monclient: monmap: 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:38:59.123375+0000 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:38:59.123375+0000 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 1 Processor -- start 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 1 -- start start 2026-02-22T21:38:59.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient: init 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient: _add_conns ranks=[2,0,1,3,5,4] 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f725c150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f725c15c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f725c0703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c15a920 con 0x7f725c15c710 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c15af60 con 0x7f725c0703b0 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c15b240 con 0x7f725c150220 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.122+0000 7f72647a9640 10 monclient(hunting): authenticate will time out at 1214.703796s 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.123+0000 7f7261d1d640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f725c15c710 0x7f725c06fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:41952/0 (socket says 10.20.193.51:41952) 2026-02-22T21:38:59.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.123+0000 7f7261d1d640 1 -- 10.20.193.51:0/2498491659 learned_addr learned my addr 10.20.193.51:0/2498491659 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500439865 0 0) 0x7f725c15a920 con 0x7f725c15c710 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _negotiate 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): my global_id is 4770 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7244003620 con 0x7f725c15c710 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959672242 0 0) 0x7f725c15b240 con 0x7f725c150220 2026-02-22T21:38:59.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _negotiate 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): my global_id is 4703 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f725c15a920 con 0x7f725c150220 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732876437 0 0) 0x7f725c15af60 con 0x7f725c0703b0 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _negotiate 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 10 monclient(hunting): my global_id is 4741 2026-02-22T21:38:59.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.124+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f725c15b240 con 0x7f725c0703b0 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2772803202 0 0) 0x7f7244003620 con 0x7f725c15c710 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 >> v1:10.20.193.125:6789/0 conn(0x7f725c150220 legacy=0x7f725c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 >> v1:10.20.193.68:6789/0 conn(0x7f725c0703b0 legacy=0x7f725c159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _finish_hunting 0 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:38:59.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c15b520 con 0x7f725c15c710 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _finish_auth 0 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _check_auth_tickets 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f72647a9640 5 monclient: authenticate success, global_id 4770 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f72647a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f7250002f00 con 0x7f725c15c710 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: dump: 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:59.140 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:59.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 >> v1:10.20.193.51:6789/0 conn(0x7f725c15c710 legacy=0x7f725c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f7244004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): picked mon.c con 0x7f7244003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7244004e40 0x7f72440052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): picked mon.a con 0x7f7244004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7244009ab0 0x7f724400bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): picked mon.b con 0x7f7244009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f724400c490 con 0x7f7244004e40 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f724400c640 con 0x7f7244009ab0 2026-02-22T21:38:59.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f724400c840 con 0x7f7244003fe0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f726151c640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.126+0000 7f72647a9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f726251e640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f7244004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f7261d1d640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7244004e40 0x7f72440052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f7262d1f640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7244009ab0 0x7f724400bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f726251e640 10 monclient(hunting): get_auth_request con 0x7f7244003fe0 auth_method 0 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f726251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f726251e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.127+0000 7f726251e640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:38:59.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): get_auth_request con 0x7f7244009ab0 auth_method 0 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): get_auth_request con 0x7f7244004e40 auth_method 0 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:38:59.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.128+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 10 monclient(hunting): handle_auth_done global_id 4770 payload 293 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 -- 10.20.193.51:0/2498491659 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7244009ab0 msgr2=0x7f724400bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7244009ab0 0x7f724400bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 -- 10.20.193.51:0/2498491659 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7244004e40 msgr2=0x7f72440052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f7244004e40 0x7f72440052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 10 monclient: _finish_hunting 0 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 monclient: found mon.c 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f7262d1f640 1 -- 10.20.193.51:0/2498491659 reap_dead start 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 1 -- 10.20.193.51:0/2498491659 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7244003620 con 0x7f7244003fe0 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 10 monclient: _finish_auth 0 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 10 monclient: _check_auth_tickets 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.129+0000 7f726251e640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:59.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.130+0000 7f726251e640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f7244004900 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f724c0031f0 tx=0x7f724c02ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95742345a1a0a43 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c002e10 con 0x7f7244003fe0 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: dump: 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:59.147 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f72647a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c0052d0 con 0x7f7244003fe0 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f72647a9640 10 monclient: get_monmap_and_config success 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 1 -- 10.20.193.51:0/2498491659 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c02f760 con 0x7f7244003fe0 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f72647a9640 4 set_mon_vals no callback set 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:38:59.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.131+0000 7f726151c640 10 monclient: dump: 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 10 monclient: shutdown 2026-02-22T21:38:59.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 -- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 msgr2=0x7f7244004900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f7244004900 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f724c0031f0 tx=0x7f724c02ef40 comp rx=0 tx=0).stop 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 -- 10.20.193.51:0/2498491659 shutdown_connections 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 --2- 10.20.193.51:0/2498491659 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f7244004900 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 -- 10.20.193.51:0/2498491659 >> 10.20.193.51:0/2498491659 conn(0x7f725c0cd7b0 msgr2=0x7f725c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 -- 10.20.193.51:0/2498491659 shutdown_connections 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 1 -- 10.20.193.51:0/2498491659 wait complete. 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 10 monclient: build_initial_monmap 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.132+0000 7f72647a9640 10 monclient: monmap: 2026-02-22T21:38:59.150 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 librados: starting msgr at 2026-02-22T21:38:59.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 librados: starting objecter 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 Processor -- start 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 -- start start 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 librados: setting wanted keys 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 librados: calling monclient init 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient: init 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient: _reopen_session rank -1 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f725c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-c con 0x7f7244003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f725c1596f0 0x7f725c06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-a con 0x7f725c1596f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:38:59.152 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): picked mon.noname-b con 0x7f725c06e3f0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f725c150600 con 0x7f725c1596f0 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f725c15bae0 con 0x7f725c06e3f0 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f725c150780 con 0x7f7244003fe0 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): _renew_subs 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f72647a9640 10 monclient(hunting): authenticate will time out at 1214.714557s 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f7262d1f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f7261d1d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f725c1596f0 0x7f725c06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.133+0000 7f726251e640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f725c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:33682/0 (socket says 10.20.193.51:33682) 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 1 -- 10.20.193.51:0/4018519545 learned_addr learned my addr 10.20.193.51:0/4018519545 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): get_auth_request con 0x7f725c06e3f0 auth_method 0 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): get_auth_request con 0x7f725c1596f0 auth_method 0 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): get_auth_request con 0x7f7244003fe0 auth_method 0 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7262d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f726251e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:59.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.134+0000 7f7261d1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 10 monclient(hunting): handle_auth_done global_id 4744 payload 824 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 -- 10.20.193.51:0/4018519545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 msgr2=0x7f725c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f725c1591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 -- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f725c1596f0 msgr2=0x7f725c06deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f725c1596f0 0x7f725c06deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 10 monclient: _finish_hunting 0 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 monclient: found mon.noname-b 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:38:59.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.135+0000 7f7262d1f640 1 -- 10.20.193.51:0/4018519545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c14ee30 con 0x7f725c06e3f0 2026-02-22T21:38:59.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.136+0000 7f7262d1f640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f725c0c4690 tx=0x7f725800f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:59.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.136+0000 7f7242ffd640 10 client.?.objecter ms_handle_connect 0x7f725c06e3f0 2026-02-22T21:38:59.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.136+0000 7f7242ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258017020 con 0x7f725c06e3f0 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: dump: 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.160 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: _finish_auth 0 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725800fe20 con 0x7f725c06e3f0 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 5 monclient: authenticate success, global_id 4744 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _renew_subs 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725c14f110 con 0x7f725c06e3f0 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258013610 con 0x7f725c06e3f0 2026-02-22T21:38:59.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7242ffd640 10 monclient: dump: 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f7260d1b640 4 set_mon_vals no callback set 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:38:59.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _renew_subs 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725c06d6b0 con 0x7f725c06e3f0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 1 librados: init done 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.138+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725c0c73a0 con 0x7f725c06e3f0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.149+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f72580137b0 con 0x7f725c06e3f0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.149+0000 7f7242ffd640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 0x7f722c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.150+0000 7f726251e640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 0x7f722c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.150+0000 7f726251e640 10 monclient: get_auth_request con 0x7f722c078e00 auth_method 0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.150+0000 7f726251e640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 0x7f722c07b2c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f724c037000 tx=0x7f724c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.150+0000 7f7242ffd640 10 client.4744.objecter ms_handle_connect 0x7f722c078e00 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.151+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f725805e8a0 con 0x7f725c06e3f0 2026-02-22T21:38:59.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.151+0000 7f7242ffd640 10 client.4744.objecter ms_dispatch2 0x7f725c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:38:59.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.151+0000 7f7242ffd640 3 client.4744.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:38:59.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.151+0000 7f7242ffd640 3 client.4744.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:38:59.164 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.151+0000 7f7242ffd640 20 client.4744.objecter dump_active .. 0 homeless 2026-02-22T21:38:59.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.251+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7258021020 con 0x7f725c06e3f0 2026-02-22T21:38:59.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.251+0000 7f7242ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:38:59.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.251+0000 7f7242ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:38:59.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.351+0000 7f72647a9640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:38:59.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.351+0000 7f72647a9640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:59.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.351+0000 7f72647a9640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:38:59.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.351+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f725c074da0 con 0x7f725c06e3f0 2026-02-22T21:38:59.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f7242ffd640 1 -- 10.20.193.51:0/4018519545 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f7258062860 con 0x7f725c06e3f0 2026-02-22T21:38:59.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f7242ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:38:59.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f7242ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:38:59.357 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:38:59.357 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 10 librados: watch_flush enter 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 10 librados: watch_flush exit 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 20 client.4744.objecter shutdown clearing up homeless session... 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 10 client.4744.objecter successfully canceled tick 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 msgr2=0x7f722c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 0x7f722c07b2c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f724c037000 tx=0x7f724c033000 comp rx=0 tx=0).stop 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 10 monclient: shutdown 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 msgr2=0x7f725c14ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f725c0c4690 tx=0x7f725800f5a0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 shutdown_connections 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f722c078e00 0x7f722c07b2c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f725c06e3f0 0x7f725c14ca30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f725c1596f0 0x7f725c06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 --2- 10.20.193.51:0/4018519545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7244003fe0 0x7f725c1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 >> 10.20.193.51:0/4018519545 conn(0x7f725c0cd7b0 msgr2=0x7f725c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 shutdown_connections 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 -- 10.20.193.51:0/4018519545 wait complete. 2026-02-22T21:38:59.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:38:59.355+0000 7f72647a9640 1 librados: shutdown 2026-02-22T21:39:01.369 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:32.523 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.398 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:32.552 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.406 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:32.560 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.426 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:32.541 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.426 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:32.577 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.427 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:32.542 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.428 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:32.579 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.434 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:32.550 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:01.435 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:32.587 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:02.367 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.433+0000 7fd114038640 10 monclient: get_monmap_and_config 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.433+0000 7fd114038640 10 monclient: build_initial_monmap 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.433+0000 7fd114038640 10 monclient: monmap: 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:02.435459+0000 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:02.435459+0000 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:02.436 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 Processor -- start 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 -- start start 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient: init 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient: _add_conns ranks=[2,5,3,4,1,0] 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd10c150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd10c15c710 0x7fd10c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): picked mon.noname-c con 0x7fd10c15c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd10c150600 0x7fd10c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): picked mon.noname-a con 0x7fd10c150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd10c15a920 con 0x7fd10c150220 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fd10c15af60 con 0x7fd10c150600 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd10c15b240 con 0x7fd10c15c710 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:02.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.434+0000 7fd114038640 10 monclient(hunting): authenticate will time out at 1218.015752s 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd10c150600 0x7fd10c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fd10c150600 0x7fd10c06fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:58114/0 (socket says 10.20.193.51:58114) 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 1 -- 10.20.193.51:0/1037922239 learned_addr learned my addr 10.20.193.51:0/1037922239 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): get_auth_request con 0x7fd10c150600 auth_method 0 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:02.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:02.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:02.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:02.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.446+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:02.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1115ac640 1 --2- 10.20.193.51:0/1037922239 >> v2:10.20.193.125:3300/0 conn(0x7fd10c15c710 0x7fd10c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 10 monclient(hunting): handle_auth_done global_id 4776 payload 274 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 1 -- 10.20.193.51:0/1037922239 >> v2:10.20.193.125:3300/0 conn(0x7fd10c15c710 msgr2=0x7fd10c0c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 1 --2- 10.20.193.51:0/1037922239 >> v2:10.20.193.125:3300/0 conn(0x7fd10c15c710 0x7fd10c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 1 -- 10.20.193.51:0/1037922239 >> v1:10.20.193.125:6789/0 conn(0x7fd10c150220 legacy=0x7fd10c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 10 monclient: _finish_hunting 0 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 1 monclient: found mon.noname-a 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.447+0000 7fd1125ae640 1 -- 10.20.193.51:0/1037922239 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c15b520 con 0x7fd10c150600 2026-02-22T21:39:02.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.449+0000 7fd1125ae640 1 --2- 10.20.193.51:0/1037922239 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150600 0x7fd10c06fca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd108009e50 tx=0x7fd10802f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e489355cc5fb7ecb server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.451+0000 7fd110dab640 1 -- 10.20.193.51:0/1037922239 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10802fd50 con 0x7fd10c150600 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.451+0000 7fd110dab640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: dump: 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: _finish_auth 0 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: _check_auth_tickets 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 5 monclient: authenticate success, global_id 4776 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 1 -- 10.20.193.51:0/1037922239 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1080373f0 con 0x7fd10c150600 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 1 -- 10.20.193.51:0/1037922239 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1080377c0 con 0x7fd10c150600 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd110dab640 10 monclient: dump: 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:02.454 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 4 set_mon_vals no callback set 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 10 monclient: shutdown 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 -- 10.20.193.51:0/1037922239 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150600 msgr2=0x7fd10c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 --2- 10.20.193.51:0/1037922239 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150600 0x7fd10c06fca0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd108009e50 tx=0x7fd10802f330 comp rx=0 tx=0).stop 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 -- 10.20.193.51:0/1037922239 shutdown_connections 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 --2- 10.20.193.51:0/1037922239 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150600 0x7fd10c06fca0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 --2- 10.20.193.51:0/1037922239 >> v2:10.20.193.125:3300/0 conn(0x7fd10c15c710 0x7fd10c0c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 -- 10.20.193.51:0/1037922239 >> 10.20.193.51:0/1037922239 conn(0x7fd10c0cd7b0 msgr2=0x7fd10c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 -- 10.20.193.51:0/1037922239 shutdown_connections 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 1 -- 10.20.193.51:0/1037922239 wait complete. 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 10 monclient: build_initial_monmap 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.452+0000 7fd114038640 10 monclient: monmap: 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:02.455 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 librados: starting msgr at 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 librados: starting objecter 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 Processor -- start 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 -- start start 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 librados: setting wanted keys 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 librados: calling monclient init 2026-02-22T21:39:02.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient: init 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): picked mon.noname-a con 0x7fd10c150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd10c150600 0x7fd10c1f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): picked mon.noname-b con 0x7fd10c150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd10c15c710 0x7fd10c1f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): picked mon.noname-c con 0x7fd10c15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c0701e0 con 0x7fd10c150220 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c15bae0 con 0x7fd10c150600 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c070360 con 0x7fd10c15c710 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd114038640 10 monclient(hunting): authenticate will time out at 1218.034617s 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd111dad640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.453+0000 7fd1125ae640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd10c15c710 0x7fd10c1f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:58124/0 (socket says 10.20.193.51:58124) 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 1 -- 10.20.193.51:0/543530902 learned_addr learned my addr 10.20.193.51:0/543530902 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): get_auth_request con 0x7fd10c150220 auth_method 0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): get_auth_request con 0x7fd10c15c710 auth_method 0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd10c150600 0x7fd10c1f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd111dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1125ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): get_auth_request con 0x7fd10c150600 auth_method 0 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:02.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.454+0000 7fd1115ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 10 monclient(hunting): handle_auth_done global_id 4779 payload 824 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 -- 10.20.193.51:0/543530902 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd10c15c710 msgr2=0x7fd10c1f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd10c15c710 0x7fd10c1f1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 -- 10.20.193.51:0/543530902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd10c150600 msgr2=0x7fd10c1f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd10c150600 0x7fd10c1f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 10 monclient: _finish_hunting 0 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 monclient: found mon.noname-a 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.455+0000 7fd111dad640 1 -- 10.20.193.51:0/543530902 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c154d80 con 0x7fd10c150220 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.456+0000 7fd111dad640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00b570 tx=0x7fd0fc00ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.456+0000 7fd0fa7fc640 10 client.?.objecter ms_handle_connect 0x7fd10c150220 2026-02-22T21:39:02.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.456+0000 7fd0fa7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc013020 con 0x7fd10c150220 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: dump: 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:02.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0fc0044a0 con 0x7fd10c150220 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc0029c0 con 0x7fd10c150220 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 5 monclient: authenticate success, global_id 4779 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fbfff640 4 set_mon_vals no callback set 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _renew_subs 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10c1571b0 con 0x7fd10c150220 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd0fa7fc640 10 monclient: dump: 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 client.4779.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _renew_subs 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10c1f4b30 con 0x7fd10c150220 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 1 librados: init done 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.458+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10c06fca0 con 0x7fd10c150220 2026-02-22T21:39:02.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.469+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd0fc002b60 con 0x7fd10c150220 2026-02-22T21:39:02.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.470+0000 7fd0fa7fc640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 0x7fd0dc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:02.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.470+0000 7fd1115ac640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 0x7fd0dc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:02.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.470+0000 7fd1115ac640 10 monclient: get_auth_request con 0x7fd0dc078e50 auth_method 0 2026-02-22T21:39:02.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.470+0000 7fd1115ac640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 0x7fd0dc07b310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd10c14d280 tx=0x7fd100008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:02.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.470+0000 7fd0fa7fc640 10 client.4779.objecter ms_handle_connect 0x7fd0dc078e50 2026-02-22T21:39:02.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd0fc05df90 con 0x7fd10c150220 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 10 client.4779.objecter ms_dispatch2 0x7fd10c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 3 client.4779.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 3 client.4779.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 20 client.4779.objecter dump_active .. 0 homeless 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd0fc01d020 con 0x7fd10c150220 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:02.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.574+0000 7fd0fa7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:02.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.676+0000 7fd114038640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:02.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.676+0000 7fd114038640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:02.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.676+0000 7fd114038640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:02.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.676+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd10c0cdc30 con 0x7fd10c150220 2026-02-22T21:39:02.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.678+0000 7fd0fa7fc640 1 -- 10.20.193.51:0/543530902 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fd0fc061f50 con 0x7fd10c150220 2026-02-22T21:39:02.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.678+0000 7fd0fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:02.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.678+0000 7fd0fa7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:02.681 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:02.681 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 10 librados: watch_flush enter 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 10 librados: watch_flush exit 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 20 client.4779.objecter shutdown clearing up homeless session... 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 10 client.4779.objecter successfully canceled tick 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 msgr2=0x7fd0dc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 0x7fd0dc07b310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd10c14d280 tx=0x7fd100008040 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 10 monclient: shutdown 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 msgr2=0x7fd10c0cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc00b570 tx=0x7fd0fc00ba40 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 shutdown_connections 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd0dc078e50 0x7fd0dc07b310 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd10c15c710 0x7fd10c1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd10c150600 0x7fd10c1f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 --2- 10.20.193.51:0/543530902 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd10c150220 0x7fd10c0cc2e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 >> 10.20.193.51:0/543530902 conn(0x7fd10c0cd7b0 msgr2=0x7fd10c15cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 shutdown_connections 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 -- 10.20.193.51:0/543530902 wait complete. 2026-02-22T21:39:02.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:02.679+0000 7fd114038640 1 librados: shutdown 2026-02-22T21:39:05.692 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 10 monclient: get_monmap_and_config 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 10 monclient: build_initial_monmap 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 10 monclient: monmap: 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:05.761853+0000 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:05.761853+0000 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:05.762 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 1 Processor -- start 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 1 -- start start 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.760+0000 7fe5bfe68640 10 monclient: init 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient: _add_conns ranks=[4,3,5,2,0,1] 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe5b81527b0 0x7fe5b80c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b81527b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe5b80bf800 0x7fe5b806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b80bf800 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe5b80bfc90 0x7fe5b80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b80bfc90 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fe5b80be810 con 0x7fe5b80bf800 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe5b8152b90 con 0x7fe5b81527b0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe5b815ffa0 con 0x7fe5b80bfc90 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bfe68640 10 monclient(hunting): authenticate will time out at 1221.342243s 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe5b81527b0 0x7fe5b80c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bd3dc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe5b80bf800 0x7fe5b806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe5b81527b0 0x7fe5b80c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:33712/0 (socket says 10.20.193.51:33712) 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 1 -- 10.20.193.51:0/631281578 learned_addr learned my addr 10.20.193.51:0/631281578 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 10 monclient(hunting): get_auth_request con 0x7fe5b81527b0 auth_method 0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5be3de640 1 --2- 10.20.193.51:0/631281578 >> v2:10.20.193.125:3300/0 conn(0x7fe5b80bfc90 0x7fe5b80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bdbdd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bd3dc640 10 monclient(hunting): get_auth_request con 0x7fe5b80bf800 auth_method 0 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bd3dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bd3dc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.761+0000 7fe5bd3dc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): get_auth_request con 0x7fe5b80bfc90 auth_method 0 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.762+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_done global_id 4750 payload 274 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 -- 10.20.193.51:0/631281578 >> v2:10.20.193.125:3300/0 conn(0x7fe5b80bfc90 msgr2=0x7fe5b80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/631281578 >> v2:10.20.193.125:3300/0 conn(0x7fe5b80bfc90 0x7fe5b80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 -- 10.20.193.51:0/631281578 >> v2:10.20.193.51:3300/0 conn(0x7fe5b80bf800 msgr2=0x7fe5b806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/631281578 >> v2:10.20.193.51:3300/0 conn(0x7fe5b80bf800 0x7fe5b806fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 10 monclient: _finish_hunting 0 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 monclient: found mon.noname-b 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:05.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.763+0000 7fe5bdbdd640 1 -- 10.20.193.51:0/631281578 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b80bea60 con 0x7fe5b81527b0 2026-02-22T21:39:05.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.764+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/631281578 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b80c9660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80c8f90 tx=0x7fe5a802ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d8b42f62dbdb5b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 1 -- 10.20.193.51:0/631281578 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a803c070 con 0x7fe5b81527b0 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: dump: 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.768 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: _finish_auth 0 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: _check_auth_tickets 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 1 -- 10.20.193.51:0/631281578 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a802fbd0 con 0x7fe5b81527b0 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bfe68640 5 monclient: authenticate success, global_id 4750 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bfe68640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 1 -- 10.20.193.51:0/631281578 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8037440 con 0x7fe5b81527b0 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bfe68640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bfe68640 4 set_mon_vals no callback set 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.765+0000 7fe5bcbdb640 10 monclient: dump: 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 10 monclient: shutdown 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 -- 10.20.193.51:0/631281578 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 msgr2=0x7fe5b80c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/631281578 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b80c9660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80c8f90 tx=0x7fe5a802ef50 comp rx=0 tx=0).stop 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 -- 10.20.193.51:0/631281578 shutdown_connections 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/631281578 >> v2:10.20.193.125:3300/0 conn(0x7fe5b80bfc90 0x7fe5b80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/631281578 >> v2:10.20.193.51:3300/0 conn(0x7fe5b80bf800 0x7fe5b806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/631281578 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b80c9660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 -- 10.20.193.51:0/631281578 >> 10.20.193.51:0/631281578 conn(0x7fe5b80cd7b0 msgr2=0x7fe5b80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 -- 10.20.193.51:0/631281578 shutdown_connections 2026-02-22T21:39:05.769 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 -- 10.20.193.51:0/631281578 wait complete. 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 10 monclient: build_initial_monmap 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 10 monclient: monmap: 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:05.770 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 librados: starting msgr at 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 librados: starting objecter 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.766+0000 7fe5bfe68640 1 Processor -- start 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 -- start start 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 librados: setting wanted keys 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 librados: calling monclient init 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient: init 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5b80bf800 0x7fe5b81f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b80bf800 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b80bfc90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b815f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b81527b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b80bf300 con 0x7fe5b80bfc90 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b80bf020 con 0x7fe5b81527b0 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b806d710 con 0x7fe5b80bf800 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bfe68640 10 monclient(hunting): authenticate will time out at 1221.348438s 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5be3de640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b815f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:58148/0 (socket says 10.20.193.51:58148) 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 1 -- 10.20.193.51:0/3402528454 learned_addr learned my addr 10.20.193.51:0/3402528454 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 10 monclient(hunting): get_auth_request con 0x7fe5b80bfc90 auth_method 0 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5bd3dc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5be3de640 10 monclient(hunting): get_auth_request con 0x7fe5b81527b0 auth_method 0 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5be3de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5be3de640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.767+0000 7fe5be3de640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5b80bf800 0x7fe5b81f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5be3de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): get_auth_request con 0x7fe5b80bf800 auth_method 0 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.768+0000 7fe5bdbdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 10 monclient(hunting): handle_auth_done global_id 4785 payload 824 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 -- 10.20.193.51:0/3402528454 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5b80bf800 msgr2=0x7fe5b81f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5b80bf800 0x7fe5b81f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 -- 10.20.193.51:0/3402528454 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 msgr2=0x7fe5b815f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:05.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b815f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 10 monclient: _finish_hunting 0 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 monclient: found mon.noname-a 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.769+0000 7fe5bd3dc640 1 -- 10.20.193.51:0/3402528454 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b815b0d0 con 0x7fe5b80bfc90 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.770+0000 7fe5bd3dc640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00e870 tx=0x7fe5ac00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.770+0000 7fe5a67fc640 10 client.?.objecter ms_handle_connect 0x7fe5b80bfc90 2026-02-22T21:39:05.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.770+0000 7fe5a67fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac017070 con 0x7fe5b80bfc90 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: dump: 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.774 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5ac00c8e0 con 0x7fe5b80bfc90 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac00ccd0 con 0x7fe5b80bfc90 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 5 monclient: authenticate success, global_id 4785 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a7fff640 4 set_mon_vals no callback set 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _renew_subs 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b815b3b0 con 0x7fe5b80bfc90 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5a67fc640 10 monclient: dump: 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 client.4785.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _renew_subs 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b815b970 con 0x7fe5b80bfc90 2026-02-22T21:39:05.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 1 librados: init done 2026-02-22T21:39:05.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:05.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:05.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:05.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.772+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5b80c9660 con 0x7fe5b80bfc90 2026-02-22T21:39:05.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.783+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe5ac013470 con 0x7fe5b80bfc90 2026-02-22T21:39:05.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.784+0000 7fe5a67fc640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 0x7fe58807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:05.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.784+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 0x7fe58807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:05.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.784+0000 7fe5bdbdd640 10 monclient: get_auth_request con 0x7fe588078e00 auth_method 0 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.784+0000 7fe5bdbdd640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 0x7fe58807b2c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80709e0 tx=0x7fe5a8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.784+0000 7fe5a67fc640 10 client.4785.objecter ms_handle_connect 0x7fe588078e00 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.785+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe5ac05e370 con 0x7fe5b80bfc90 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.785+0000 7fe5a67fc640 10 client.4785.objecter ms_dispatch2 0x7fe5b8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.785+0000 7fe5a67fc640 3 client.4785.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.785+0000 7fe5a67fc640 3 client.4785.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:05.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.785+0000 7fe5a67fc640 20 client.4785.objecter dump_active .. 0 homeless 2026-02-22T21:39:05.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.883+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe5ac062330 con 0x7fe5b80bfc90 2026-02-22T21:39:05.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.883+0000 7fe5a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:05.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.883+0000 7fe5a67fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:05.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.985+0000 7fe5bfe68640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:05.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.985+0000 7fe5bfe68640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:05.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.985+0000 7fe5bfe68640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:05.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.985+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe5b8156060 con 0x7fe5b80bfc90 2026-02-22T21:39:05.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.987+0000 7fe5a67fc640 1 -- 10.20.193.51:0/3402528454 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe5ac066630 con 0x7fe5b80bfc90 2026-02-22T21:39:05.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.987+0000 7fe5a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:05.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.987+0000 7fe5a67fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:05.990 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:05.990 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 10 librados: watch_flush enter 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 10 librados: watch_flush exit 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 20 client.4785.objecter shutdown clearing up homeless session... 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 10 client.4785.objecter successfully canceled tick 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 msgr2=0x7fe58807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:05.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 0x7fe58807b2c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80709e0 tx=0x7fe5a8033000 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 10 monclient: shutdown 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 msgr2=0x7fe5b815e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00e870 tx=0x7fe5ac00ed40 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 shutdown_connections 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe588078e00 0x7fe58807b2c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5b81527b0 0x7fe5b815f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5b80bfc90 0x7fe5b815e830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 --2- 10.20.193.51:0/3402528454 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5b80bf800 0x7fe5b81f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 >> 10.20.193.51:0/3402528454 conn(0x7fe5b80cd7b0 msgr2=0x7fe5b80c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 shutdown_connections 2026-02-22T21:39:05.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 -- 10.20.193.51:0/3402528454 wait complete. 2026-02-22T21:39:05.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:05.988+0000 7fe5bfe68640 1 librados: shutdown 2026-02-22T21:39:09.000 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:09.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 10 monclient: get_monmap_and_config 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 10 monclient: build_initial_monmap 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 10 monclient: monmap: 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:09.069150+0000 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:09.069150+0000 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 1 Processor -- start 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 1 -- start start 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.067+0000 7f4763cea640 10 monclient: init 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient: _add_conns ranks=[1,0,2,4,5,3] 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f475c152ab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f475c06d9d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f475c0c95a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f475c06f1d0 con 0x7f475c06d9d0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f475c06de50 con 0x7f475c152ab0 2026-02-22T21:39:09.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f475c06e130 con 0x7f475c0c95a0 2026-02-22T21:39:09.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:09.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f4763cea640 10 monclient(hunting): authenticate will time out at 1224.649475s 2026-02-22T21:39:09.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f476125e640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f475c06d9d0 0x7f475c0c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58134/0 (socket says 10.20.193.51:58134) 2026-02-22T21:39:09.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.068+0000 7f476125e640 1 -- 10.20.193.51:0/2187340112 learned_addr learned my addr 10.20.193.51:0/2187340112 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1191510430 0 0) 0x7f475c06f1d0 con 0x7f475c06d9d0 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _negotiate 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): my global_id is 4788 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4744003620 con 0x7f475c06d9d0 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644088171 0 0) 0x7f475c06e130 con 0x7f475c0c95a0 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _negotiate 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 10 monclient(hunting): my global_id is 4718 2026-02-22T21:39:09.073 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.070+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f475c06f1d0 con 0x7f475c0c95a0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237396401 0 0) 0x7f475c06de50 con 0x7f475c152ab0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 10 monclient(hunting): _negotiate 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 10 monclient(hunting): my global_id is 4756 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.071+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f475c06e130 con 0x7f475c152ab0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1220059829 0 0) 0x7f4744003620 con 0x7f475c06d9d0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 >> v1:10.20.193.125:6789/0 conn(0x7f475c0c95a0 legacy=0x7f475c1632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 >> v1:10.20.193.68:6789/0 conn(0x7f475c152ab0 legacy=0x7f475c0c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _finish_hunting 0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:39:09.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475c06e410 con 0x7f475c06d9d0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _finish_auth 0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _check_auth_tickets 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f4750002f00 con 0x7f475c06d9d0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: dump: 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 >> v1:10.20.193.51:6789/0 conn(0x7f475c06d9d0 legacy=0x7f475c0c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744003fe0 0x7f4744004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): picked mon.b con 0x7f4744003fe0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 0x7f47440052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): picked mon.a con 0x7f4744004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4744009ab0 0x7f474400bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): picked mon.c con 0x7f4744009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f474400c490 con 0x7f4744004e40 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f474400c640 con 0x7f4744003fe0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f474400c840 con 0x7f4744009ab0 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:09.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.072+0000 7f4760a5d640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f4762260640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4744009ab0 0x7f474400bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f476125e640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 0x7f47440052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f4762260640 10 monclient(hunting): get_auth_request con 0x7f4744009ab0 auth_method 0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f4762260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f4762260640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f4762260640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f476125e640 10 monclient(hunting): get_auth_request con 0x7f4744004e40 auth_method 0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f476125e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f476125e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.073+0000 7f476125e640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744003fe0 0x7f4744004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): get_auth_request con 0x7f4744003fe0 auth_method 0 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.074+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 10 monclient(hunting): handle_auth_done global_id 4788 payload 293 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 -- 10.20.193.51:0/2187340112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4744009ab0 msgr2=0x7f474400bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4744009ab0 0x7f474400bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 -- 10.20.193.51:0/2187340112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744003fe0 msgr2=0x7f4744004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744003fe0 0x7f4744004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 10 monclient: _finish_hunting 0 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 monclient: found mon.a 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f4762260640 1 -- 10.20.193.51:0/2187340112 reap_dead start 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 1 -- 10.20.193.51:0/2187340112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4744003620 con 0x7f4744004e40 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 10 monclient: _finish_auth 0 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 10 monclient: _check_auth_tickets 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f476125e640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f4763cea640 5 monclient: authenticate success, global_id 4788 2026-02-22T21:39:09.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.075+0000 7f4763cea640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:09.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.076+0000 7f476125e640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 0x7f47440052c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f475c0c43e0 tx=0x7f475002f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61af6066fb3395d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:09.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750007d00 con 0x7f4744004e40 2026-02-22T21:39:09.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:09.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:09.080 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: dump: 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4763cea640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4750033030 con 0x7f4744004e40 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 1 -- 10.20.193.51:0/2187340112 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750005c10 con 0x7f4744004e40 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4760a5d640 10 monclient: dump: 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.081 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4763cea640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.077+0000 7f4763cea640 4 set_mon_vals no callback set 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 10 monclient: shutdown 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 -- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 msgr2=0x7f47440052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 0x7f47440052c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f475c0c43e0 tx=0x7f475002f490 comp rx=0 tx=0).stop 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 -- 10.20.193.51:0/2187340112 shutdown_connections 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 --2- 10.20.193.51:0/2187340112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4744004e40 0x7f47440052c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 -- 10.20.193.51:0/2187340112 >> 10.20.193.51:0/2187340112 conn(0x7f475c0cd7b0 msgr2=0x7f475c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 -- 10.20.193.51:0/2187340112 shutdown_connections 2026-02-22T21:39:09.082 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 1 -- 10.20.193.51:0/2187340112 wait complete. 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 10 monclient: build_initial_monmap 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.078+0000 7f4763cea640 10 monclient: monmap: 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 librados: starting msgr at 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 librados: starting objecter 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 Processor -- start 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 -- start start 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 librados: setting wanted keys 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 librados: calling monclient init 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient: init 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 0x7f475c073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-b con 0x7f4744004e40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f475c15b0e0 0x7f475c15b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-c con 0x7f475c15b0e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 0x7f475c0c04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): picked mon.noname-a con 0x7f475c0c0020 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f475c06e9d0 con 0x7f475c0c0020 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f475c06ed40 con 0x7f4744004e40 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f475c152e90 con 0x7f475c15b0e0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.079+0000 7f4763cea640 10 monclient(hunting): authenticate will time out at 1224.660839s 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 0x7f475c0c04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 0x7f475c073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 0x7f475c0c04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48254/0 (socket says 10.20.193.51:48254) 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 1 -- 10.20.193.51:0/3219948919 learned_addr learned my addr 10.20.193.51:0/3219948919 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): get_auth_request con 0x7f475c0c0020 auth_method 0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): get_auth_request con 0x7f4744004e40 auth_method 0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f475c15b0e0 0x7f475c15b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4762260640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f4761a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): get_auth_request con 0x7f475c15b0e0 auth_method 0 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:09.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.080+0000 7f476125e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 10 monclient(hunting): handle_auth_done global_id 4759 payload 824 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 -- 10.20.193.51:0/3219948919 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f475c15b0e0 msgr2=0x7f475c15b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f475c15b0e0 0x7f475c15b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 -- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 msgr2=0x7f475c0c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 0x7f475c0c04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 10 monclient: _finish_hunting 0 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 monclient: found mon.noname-b 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.081+0000 7f4761a5f640 1 -- 10.20.193.51:0/3219948919 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475c15baa0 con 0x7f4744004e40 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.082+0000 7f4761a5f640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 0x7f475c073dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f474c00eb90 tx=0x7f474c00c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.082+0000 7f474a7fc640 10 client.?.objecter ms_handle_connect 0x7f4744004e40 2026-02-22T21:39:09.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.082+0000 7f474a7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c018070 con 0x7f4744004e40 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: dump: 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:09.087 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f474c00cad0 con 0x7f4744004e40 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c013420 con 0x7f4744004e40 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474a7fc640 10 monclient: dump: 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f474bfff640 4 set_mon_vals no callback set 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 5 monclient: authenticate success, global_id 4759 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 10 monclient: _renew_subs 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:09.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475c0c09e0 con 0x7f4744004e40 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 10 client.4759.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 10 monclient: _renew_subs 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475c1ea2a0 con 0x7f4744004e40 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.084+0000 7f4763cea640 1 librados: init done 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.085+0000 7f4763cea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.085+0000 7f4763cea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.085+0000 7f4763cea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:09.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.085+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475c0c8e90 con 0x7f4744004e40 2026-02-22T21:39:09.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.098+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f474c01c3b0 con 0x7f4744004e40 2026-02-22T21:39:09.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.098+0000 7f474a7fc640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 0x7f472c07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:09.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f476125e640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 0x7f472c07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:09.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f476125e640 10 monclient: get_auth_request con 0x7f472c078e30 auth_method 0 2026-02-22T21:39:09.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f474a7fc640 10 client.4759.objecter ms_handle_connect 0x7f472c078e30 2026-02-22T21:39:09.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f476125e640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 0x7f472c07b2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4750036000 tx=0x7f4750007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:09.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f474c05d740 con 0x7f4744004e40 2026-02-22T21:39:09.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f474a7fc640 10 client.4759.objecter ms_dispatch2 0x7f475c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:09.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f474a7fc640 3 client.4759.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:09.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.099+0000 7f474a7fc640 3 client.4759.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:09.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.100+0000 7f474a7fc640 20 client.4759.objecter dump_active .. 0 homeless 2026-02-22T21:39:09.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.198+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f474c022020 con 0x7f4744004e40 2026-02-22T21:39:09.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.198+0000 7f474a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:09.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.198+0000 7f474a7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:09.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.299+0000 7f4763cea640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:09.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.299+0000 7f4763cea640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:09.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.299+0000 7f4763cea640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:09.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.299+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f475c1ea580 con 0x7f4744004e40 2026-02-22T21:39:09.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.301+0000 7f474a7fc640 1 -- 10.20.193.51:0/3219948919 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f475c15baa0 con 0x7f4744004e40 2026-02-22T21:39:09.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.301+0000 7f474a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:09.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.301+0000 7f474a7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:09.304 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:09.304 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 10 librados: watch_flush enter 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 10 librados: watch_flush exit 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 20 client.4759.objecter shutdown clearing up homeless session... 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 10 client.4759.objecter successfully canceled tick 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 msgr2=0x7f472c07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 0x7f472c07b2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4750036000 tx=0x7f4750007000 comp rx=0 tx=0).stop 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 10 monclient: shutdown 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 msgr2=0x7f475c073dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:09.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 0x7f475c073dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f474c00eb90 tx=0x7f474c00c4b0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 shutdown_connections 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f472c078e30 0x7f472c07b2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f475c0c0020 0x7f475c0c04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f475c15b0e0 0x7f475c15b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 --2- 10.20.193.51:0/3219948919 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4744004e40 0x7f475c073dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 >> 10.20.193.51:0/3219948919 conn(0x7f475c0cd7b0 msgr2=0x7f475c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 shutdown_connections 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 -- 10.20.193.51:0/3219948919 wait complete. 2026-02-22T21:39:09.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:09.302+0000 7f4763cea640 1 librados: shutdown 2026-02-22T21:39:12.315 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:12.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 10 monclient: get_monmap_and_config 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 10 monclient: build_initial_monmap 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 10 monclient: monmap: 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:12.384123+0000 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:12.384123+0000 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 1 Processor -- start 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 1 -- start start 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.382+0000 7f96e29e4640 10 monclient: init 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:12.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient: _add_conns ranks=[0,3,4,2,5,1] 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96dc1527b0 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f96dc0bf800 0x7f96dc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-a con 0x7f96dc0bf800 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f96dc0bfc90 0x7f96dc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-b con 0x7f96dc0bfc90 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96dc15ffa0 con 0x7f96dc1527b0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc0be810 con 0x7f96dc0bf800 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc152b90 con 0x7f96dc0bfc90 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e29e4640 10 monclient(hunting): authenticate will time out at 1227.964465s 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96dbfff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f96dc1527b0 0x7f96dc0c9660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58146/0 (socket says 10.20.193.51:58146) 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96dbfff640 1 -- 10.20.193.51:0/961483784 learned_addr learned my addr 10.20.193.51:0/961483784 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 1 --2- 10.20.193.51:0/961483784 >> v2:10.20.193.68:3300/0 conn(0x7f96dc0bfc90 0x7f96dc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): get_auth_request con 0x7f96dc0bfc90 auth_method 0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96db7fe640 1 --2- 10.20.193.51:0/961483784 >> v2:10.20.193.51:3300/0 conn(0x7f96dc0bf800 0x7f96dc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96db7fe640 10 monclient(hunting): get_auth_request con 0x7f96dc0bf800 auth_method 0 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:12.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96db7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.384+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:12.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_done global_id 4762 payload 274 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 1 -- 10.20.193.51:0/961483784 >> v2:10.20.193.51:3300/0 conn(0x7f96dc0bf800 msgr2=0x7f96dc06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 1 --2- 10.20.193.51:0/961483784 >> v2:10.20.193.51:3300/0 conn(0x7f96dc0bf800 0x7f96dc06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 1 -- 10.20.193.51:0/961483784 >> v1:10.20.193.51:6789/0 conn(0x7f96dc1527b0 legacy=0x7f96dc0c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 10 monclient: _finish_hunting 0 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 1 monclient: found mon.noname-b 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.386+0000 7f96e0f5a640 1 -- 10.20.193.51:0/961483784 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc0bea60 con 0x7f96dc0bfc90 2026-02-22T21:39:12.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.387+0000 7f96e0f5a640 1 --2- 10.20.193.51:0/961483784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc0701e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96dc156280 tx=0x7f96d002f250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f567cbbf0d4f633 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 1 -- 10.20.193.51:0/961483784 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d003c070 con 0x7f96dc0bfc90 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: dump: 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:12.391 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: _finish_auth 0 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: _check_auth_tickets 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 1 -- 10.20.193.51:0/961483784 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d0037440 con 0x7f96dc0bfc90 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 5 monclient: authenticate success, global_id 4762 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 1 -- 10.20.193.51:0/961483784 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0037810 con 0x7f96dc0bfc90 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 4 set_mon_vals no callback set 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96daffd640 10 monclient: dump: 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 10 monclient: shutdown 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 -- 10.20.193.51:0/961483784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 msgr2=0x7f96dc0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 --2- 10.20.193.51:0/961483784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc0701e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96dc156280 tx=0x7f96d002f250 comp rx=0 tx=0).stop 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 -- 10.20.193.51:0/961483784 shutdown_connections 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 --2- 10.20.193.51:0/961483784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc0701e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 --2- 10.20.193.51:0/961483784 >> v2:10.20.193.51:3300/0 conn(0x7f96dc0bf800 0x7f96dc06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 -- 10.20.193.51:0/961483784 >> 10.20.193.51:0/961483784 conn(0x7f96dc0cd7b0 msgr2=0x7f96dc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 -- 10.20.193.51:0/961483784 shutdown_connections 2026-02-22T21:39:12.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 1 -- 10.20.193.51:0/961483784 wait complete. 2026-02-22T21:39:12.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 10 monclient: build_initial_monmap 2026-02-22T21:39:12.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.389+0000 7f96e29e4640 10 monclient: monmap: 2026-02-22T21:39:12.393 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:12.393 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 librados: starting msgr at 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 librados: starting objecter 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 Processor -- start 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 -- start start 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 librados: setting wanted keys 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 librados: calling monclient init 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient: init 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f96dc0bf800 0x7f96dc1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-c con 0x7f96dc0bf800 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-b con 0x7f96dc0bfc90 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 0x7f96dc1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): picked mon.noname-a con 0x7f96dc1527b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0bf300 con 0x7f96dc1527b0 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc0bf020 con 0x7f96dc0bfc90 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc06d710 con 0x7f96dc0bf800 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e29e4640 10 monclient(hunting): authenticate will time out at 1227.971625s 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.390+0000 7f96e0f5a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 0x7f96dc1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 0x7f96dc1503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48264/0 (socket says 10.20.193.51:48264) 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 1 -- 10.20.193.51:0/3423457958 learned_addr learned my addr 10.20.193.51:0/3423457958 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): get_auth_request con 0x7f96dc1527b0 auth_method 0 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:12.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f96dc0bf800 0x7f96dc1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): get_auth_request con 0x7f96dc0bfc90 auth_method 0 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96e0f5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): get_auth_request con 0x7f96dc0bf800 auth_method 0 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.392+0000 7f96dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 10 monclient(hunting): handle_auth_done global_id 4765 payload 824 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 -- 10.20.193.51:0/3423457958 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f96dc0bf800 msgr2=0x7f96dc1f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f96dc0bf800 0x7f96dc1f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 -- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 msgr2=0x7f96dc1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 0x7f96dc1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 monclient: found mon.noname-b 2026-02-22T21:39:12.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:12.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.393+0000 7f96db7fe640 1 -- 10.20.193.51:0/3423457958 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc072b90 con 0x7f96dc0bfc90 2026-02-22T21:39:12.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.395+0000 7f96db7fe640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc1f3a70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00e870 tx=0x7f96cc00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:12.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.395+0000 7f96d8ff9640 10 client.?.objecter ms_handle_connect 0x7f96dc0bfc90 2026-02-22T21:39:12.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.395+0000 7f96d8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc017070 con 0x7f96dc0bfc90 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: dump: 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96cc00c940 con 0x7f96dc0bfc90 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 5 monclient: authenticate success, global_id 4765 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 10 monclient: _renew_subs 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc1f4420 con 0x7f96dc0bfc90 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:12.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc00cd10 con 0x7f96dc0bfc90 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96d8ff9640 10 monclient: dump: 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96da7fc640 4 set_mon_vals no callback set 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 10 client.4765.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 10 monclient: _renew_subs 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc072e30 con 0x7f96dc0bfc90 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.396+0000 7f96e29e4640 1 librados: init done 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.397+0000 7f96e29e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.397+0000 7f96e29e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.397+0000 7f96e29e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.397+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96dc0c9660 con 0x7f96dc0bfc90 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.408+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f96cc024060 con 0x7f96dc0bfc90 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.409+0000 7f96d8ff9640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 0x7f96ac07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.409+0000 7f96dbfff640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 0x7f96ac07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.409+0000 7f96dbfff640 10 monclient: get_auth_request con 0x7f96ac078e80 auth_method 0 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.409+0000 7f96dbfff640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 0x7f96ac07b340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f96c8002770 tx=0x7f96c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:12.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.409+0000 7f96d8ff9640 10 client.4765.objecter ms_handle_connect 0x7f96ac078e80 2026-02-22T21:39:12.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.411+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f96cc05df70 con 0x7f96dc0bfc90 2026-02-22T21:39:12.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.411+0000 7f96d8ff9640 10 client.4765.objecter ms_dispatch2 0x7f96dc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:12.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.411+0000 7f96d8ff9640 3 client.4765.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:12.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.411+0000 7f96d8ff9640 3 client.4765.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:12.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.411+0000 7f96d8ff9640 20 client.4765.objecter dump_active .. 0 homeless 2026-02-22T21:39:12.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.513+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f96cc061f30 con 0x7f96dc0bfc90 2026-02-22T21:39:12.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.513+0000 7f96d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:12.514 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.513+0000 7f96d8ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:12.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.613+0000 7f96e29e4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:12.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.613+0000 7f96e29e4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:12.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.613+0000 7f96e29e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:12.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.613+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f96dc0c77f0 con 0x7f96dc0bfc90 2026-02-22T21:39:12.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96d8ff9640 1 -- 10.20.193.51:0/3423457958 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f96cc066230 con 0x7f96dc0bfc90 2026-02-22T21:39:12.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:12.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96d8ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:12.618 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:12.618 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:12.619 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 10 librados: watch_flush enter 2026-02-22T21:39:12.619 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 10 librados: watch_flush exit 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 20 client.4765.objecter shutdown clearing up homeless session... 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 10 client.4765.objecter successfully canceled tick 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 msgr2=0x7f96ac07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.616+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 0x7f96ac07b340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f96c8002770 tx=0x7f96c8009040 comp rx=0 tx=0).stop 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 10 monclient: shutdown 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 msgr2=0x7f96dc1f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc1f3a70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00e870 tx=0x7f96cc00ed40 comp rx=0 tx=0).stop 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 shutdown_connections 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f96ac078e80 0x7f96ac07b340 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f96dc1527b0 0x7f96dc1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f96dc0bfc90 0x7f96dc1f3a70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 --2- 10.20.193.51:0/3423457958 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f96dc0bf800 0x7f96dc1f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 >> 10.20.193.51:0/3423457958 conn(0x7f96dc0cd7b0 msgr2=0x7f96dc0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 shutdown_connections 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 -- 10.20.193.51:0/3423457958 wait complete. 2026-02-22T21:39:12.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:12.617+0000 7f96e29e4640 1 librados: shutdown 2026-02-22T21:39:15.627 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:15.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.693+0000 7fb78e1e4640 10 monclient: get_monmap_and_config 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.693+0000 7fb78e1e4640 10 monclient: build_initial_monmap 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.693+0000 7fb78e1e4640 10 monclient: monmap: 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:15.694419+0000 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:15.694419+0000 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:15.696 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 1 Processor -- start 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 1 -- start start 2026-02-22T21:39:15.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient: init 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient: _add_conns ranks=[1,0,2,3,4,5] 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb788150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb78815c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7880703b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:15.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78815a920 con 0x7fb78815c710 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78815af60 con 0x7fb788150220 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb78815b240 con 0x7fb7880703b0 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.694+0000 7fb78e1e4640 10 monclient(hunting): authenticate will time out at 1231.274711s 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.695+0000 7fb7877fe640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fb788150220 0x7fb78815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:53044/0 (socket says 10.20.193.51:53044) 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.695+0000 7fb7877fe640 1 -- 10.20.193.51:0/3608611351 learned_addr learned my addr 10.20.193.51:0/3608611351 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.695+0000 7fb786ffd640 1 --1- 10.20.193.51:0/3608611351 >> v1:10.20.193.51:6789/0 conn(0x7fb78815c710 0x7fb78806fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58162/0 (socket says 10.20.193.51:58162) 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 160944465 0 0) 0x7fb78815a920 con 0x7fb78815c710 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _negotiate 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): my global_id is 4803 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb76c003620 con 0x7fb78815c710 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805535563 0 0) 0x7fb78815af60 con 0x7fb788150220 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _negotiate 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 10 monclient(hunting): my global_id is 4768 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.696+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb78815a920 con 0x7fb788150220 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1612813878 0 0) 0x7fb78815b240 con 0x7fb7880703b0 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 10 monclient(hunting): _negotiate 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 10 monclient(hunting): my global_id is 4727 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.697+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb78815af60 con 0x7fb7880703b0 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2914174696 0 0) 0x7fb76c003620 con 0x7fb78815c710 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 >> v1:10.20.193.125:6789/0 conn(0x7fb7880703b0 legacy=0x7fb788159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 >> v1:10.20.193.68:6789/0 conn(0x7fb788150220 legacy=0x7fb78815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _finish_hunting 0 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78815b520 con 0x7fb78815c710 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:15.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fb778002f00 con 0x7fb78815c710 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: dump: 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:15.707 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 >> v1:10.20.193.51:6789/0 conn(0x7fb78815c710 legacy=0x7fb78806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c003fe0 0x7fb76c004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): picked mon.c con 0x7fb76c003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb76c004e40 0x7fb76c0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): picked mon.a con 0x7fb76c004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 0x7fb76c00bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): picked mon.b con 0x7fb76c009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c00c490 con 0x7fb76c004e40 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c00c640 con 0x7fb76c009ab0 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c00c840 con 0x7fb76c003fe0 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.698+0000 7fb7867fc640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 0x7fb76c00bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): get_auth_request con 0x7fb76c009ab0 auth_method 0 2026-02-22T21:39:15.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb76c004e40 0x7fb76c0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): get_auth_request con 0x7fb76c004e40 auth_method 0 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.699+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c003fe0 0x7fb76c004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): get_auth_request con 0x7fb76c003fe0 auth_method 0 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 10 monclient(hunting): handle_auth_done global_id 4803 payload 293 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 -- 10.20.193.51:0/3608611351 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c003fe0 msgr2=0x7fb76c004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c003fe0 0x7fb76c004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 -- 10.20.193.51:0/3608611351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb76c004e40 msgr2=0x7fb76c0052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:15.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb76c004e40 0x7fb76c0052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 monclient: found mon.b 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 1 -- 10.20.193.51:0/3608611351 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c003620 con 0x7fb76c009ab0 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 10 monclient: _finish_auth 0 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 10 monclient: _check_auth_tickets 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb78e1e4640 5 monclient: authenticate success, global_id 4803 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb78e1e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.700+0000 7fb786ffd640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb76c004e40 0x7fb76c0052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:39:15.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.701+0000 7fb787fff640 1 -- 10.20.193.51:0/3608611351 reap_dead start 2026-02-22T21:39:15.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.701+0000 7fb787fff640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 0x7fb76c00bea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c2f0 tx=0x7fb77c02f5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f16f79edd8701c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:15.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778007c60 con 0x7fb76c009ab0 2026-02-22T21:39:15.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:15.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:15.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: dump: 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:15.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77c003c70 con 0x7fb76c009ab0 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 4 set_mon_vals no callback set 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 1 -- 10.20.193.51:0/3608611351 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c003860 con 0x7fb76c009ab0 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb7867fc640 10 monclient: dump: 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:15.721 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 10 monclient: shutdown 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 -- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 msgr2=0x7fb76c00bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 0x7fb76c00bea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c2f0 tx=0x7fb77c02f5e0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 -- 10.20.193.51:0/3608611351 shutdown_connections 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/3608611351 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb76c009ab0 0x7fb76c00bea0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 -- 10.20.193.51:0/3608611351 >> 10.20.193.51:0/3608611351 conn(0x7fb7880cd7b0 msgr2=0x7fb7880cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:15.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 -- 10.20.193.51:0/3608611351 shutdown_connections 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.703+0000 7fb78e1e4640 1 -- 10.20.193.51:0/3608611351 wait complete. 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 10 monclient: build_initial_monmap 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 10 monclient: monmap: 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:15.723 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 librados: starting msgr at 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 librados: starting objecter 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 Processor -- start 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 -- start start 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 librados: setting wanted keys 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 1 librados: calling monclient init 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.704+0000 7fb78e1e4640 10 monclient: init 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c009ab0 0x7fb7881e4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-c con 0x7fb76c009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:15.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-b con 0x7fb7880c8950 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb78814c8e0 0x7fb788073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): picked mon.noname-a con 0x7fb78814c8e0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb788150600 con 0x7fb78814c8e0 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb78815bae0 con 0x7fb7880c8950 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb788150780 con 0x7fb76c009ab0 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb78e1e4640 10 monclient(hunting): authenticate will time out at 1231.285067s 2026-02-22T21:39:15.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb78814c8e0 0x7fb788073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:46734/0 (socket says 10.20.193.51:46734) 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 1 -- 10.20.193.51:0/1034540097 learned_addr learned my addr 10.20.193.51:0/1034540097 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): get_auth_request con 0x7fb7880c8950 auth_method 0 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb7877fe640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c009ab0 0x7fb7881e4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): get_auth_request con 0x7fb78814c8e0 auth_method 0 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.726 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb786ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb787fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb7877fe640 10 monclient(hunting): get_auth_request con 0x7fb76c009ab0 auth_method 0 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb7877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb7877fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.705+0000 7fb7877fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.706+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.706+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:15.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.706+0000 7fb7877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.706+0000 7fb786ffd640 10 monclient(hunting): handle_auth_done global_id 4771 payload 824 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 -- 10.20.193.51:0/1034540097 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c009ab0 msgr2=0x7fb7881e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c009ab0 0x7fb7881e4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 -- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb78814c8e0 msgr2=0x7fb788073450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb78814c8e0 0x7fb788073450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 monclient: found mon.noname-b 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.707+0000 7fb786ffd640 1 -- 10.20.193.51:0/1034540097 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7881f4910 con 0x7fb7880c8950 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb786ffd640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb778007340 tx=0x7fb778007810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 client.?.objecter ms_handle_connect 0x7fb7880c8950 2026-02-22T21:39:15.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77800f9c0 con 0x7fb7880c8950 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 monclient: dump: 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:15.729 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 monclient: _finish_auth 0 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 10 monclient: _check_auth_tickets 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb763fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 5 monclient: authenticate success, global_id 4771 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 10 monclient: _renew_subs 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7880c7b90 con 0x7fb7880c8950 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 10 client.4771.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 10 monclient: _renew_subs 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7880c8150 con 0x7fb7880c8950 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.708+0000 7fb78e1e4640 1 librados: init done 2026-02-22T21:39:15.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb78e1e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb78e1e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb78e1e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7880c73a0 con 0x7fb7880c8950 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb778018070 con 0x7fb7880c8950 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77800fd80 con 0x7fb7880c8950 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb785ffb640 4 set_mon_vals no callback set 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.709+0000 7fb763fff640 10 monclient: dump: 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.731 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:15.732 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.817+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb77801d460 con 0x7fb7880c8950 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.818+0000 7fb763fff640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 0x7fb75807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.818+0000 7fb7877fe640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 0x7fb75807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.818+0000 7fb7877fe640 10 monclient: get_auth_request con 0x7fb758078e30 auth_method 0 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.818+0000 7fb763fff640 10 client.4771.objecter ms_handle_connect 0x7fb758078e30 2026-02-22T21:39:15.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.818+0000 7fb7877fe640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 0x7fb75807b2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb774004f00 tx=0x7fb774007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:15.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.820+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb77805e2e0 con 0x7fb7880c8950 2026-02-22T21:39:15.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.820+0000 7fb763fff640 10 client.4771.objecter ms_dispatch2 0x7fb788000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:15.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.820+0000 7fb763fff640 3 client.4771.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:15.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.820+0000 7fb763fff640 3 client.4771.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:15.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.820+0000 7fb763fff640 20 client.4771.objecter dump_active .. 0 homeless 2026-02-22T21:39:15.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.827+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb7780629a0 con 0x7fb7880c8950 2026-02-22T21:39:15.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.827+0000 7fb763fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:15.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.827+0000 7fb763fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:15.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.925+0000 7fb78e1e4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:15.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.925+0000 7fb78e1e4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:15.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.925+0000 7fb78e1e4640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:15.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.925+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb78815e0d0 con 0x7fb7880c8950 2026-02-22T21:39:15.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.929+0000 7fb763fff640 1 -- 10.20.193.51:0/1034540097 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb7780622a0 con 0x7fb7880c8950 2026-02-22T21:39:15.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.929+0000 7fb763fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:15.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.929+0000 7fb763fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:15.930 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:15.930 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:15.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 10 librados: watch_flush enter 2026-02-22T21:39:15.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 10 librados: watch_flush exit 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 20 client.4771.objecter shutdown clearing up homeless session... 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 10 client.4771.objecter successfully canceled tick 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 msgr2=0x7fb75807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 0x7fb75807b2f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb774004f00 tx=0x7fb774007e90 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 10 monclient: shutdown 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 msgr2=0x7fb7881f4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb778007340 tx=0x7fb778007810 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 shutdown_connections 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb758078e30 0x7fb75807b2f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb78814c8e0 0x7fb788073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb7880c8950 0x7fb7881f4f90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 --2- 10.20.193.51:0/1034540097 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb76c009ab0 0x7fb7881e4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 >> 10.20.193.51:0/1034540097 conn(0x7fb7880cd7b0 msgr2=0x7fb7880cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 shutdown_connections 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 -- 10.20.193.51:0/1034540097 wait complete. 2026-02-22T21:39:15.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:15.930+0000 7fb78e1e4640 1 librados: shutdown 2026-02-22T21:39:18.940 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:19.008 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.007+0000 7f1446c98640 10 monclient: get_monmap_and_config 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.007+0000 7f1446c98640 10 monclient: build_initial_monmap 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.007+0000 7f1446c98640 10 monclient: monmap: 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:19.008229+0000 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:19.008229+0000 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.007+0000 7f1446c98640 1 Processor -- start 2026-02-22T21:39:19.009 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 1 -- start start 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient: init 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient: _add_conns ranks=[0,2,1,5,3,4] 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1440150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f144015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f14400703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144015a920 con 0x7f1440150220 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144015af60 con 0x7f14400703b0 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144015b240 con 0x7f144015c710 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1446c98640 10 monclient(hunting): authenticate will time out at 1234.588550s 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1444a0d640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f1440150220 0x7f144015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:33936/0 (socket says 10.20.193.51:33936) 2026-02-22T21:39:19.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.008+0000 7f1444a0d640 1 -- 10.20.193.51:0/2858302160 learned_addr learned my addr 10.20.193.51:0/2858302160 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491868658 0 0) 0x7f144015a920 con 0x7f1440150220 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _negotiate 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): my global_id is 4809 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f142c003620 con 0x7f1440150220 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005411885 0 0) 0x7f144015af60 con 0x7f14400703b0 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _negotiate 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): my global_id is 4774 2026-02-22T21:39:19.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f144015a920 con 0x7f14400703b0 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940870386 0 0) 0x7f144015b240 con 0x7f144015c710 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _negotiate 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 10 monclient(hunting): my global_id is 4733 2026-02-22T21:39:19.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.010+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f144015af60 con 0x7f144015c710 2026-02-22T21:39:19.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1799182441 0 0) 0x7f144015a920 con 0x7f14400703b0 2026-02-22T21:39:19.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 >> v1:10.20.193.125:6789/0 conn(0x7f144015c710 legacy=0x7f144006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 >> v1:10.20.193.51:6789/0 conn(0x7f1440150220 legacy=0x7f144015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: _finish_hunting 0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144015b520 con 0x7f14400703b0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: _finish_auth 0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: _check_auth_tickets 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f143c0031e0 con 0x7f14400703b0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.011+0000 7f14377fe640 10 monclient: dump: 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 >> v1:10.20.193.68:6789/0 conn(0x7f14400703b0 legacy=0x7f1440159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c005020 0x7f142c0038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): picked mon.b con 0x7f142c005020 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f142c005530 0x7f142c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): picked mon.a con 0x7f142c005530 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 0x7f142c00bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): picked mon.c con 0x7f142c009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f142c00c490 con 0x7f142c005530 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f142c00c610 con 0x7f142c005020 2026-02-22T21:39:19.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f142c00c810 con 0x7f142c009ab0 2026-02-22T21:39:19.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:19.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f14377fe640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.012+0000 7f144520e640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 0x7f142c00bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): get_auth_request con 0x7f142c009ab0 auth_method 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f142c005530 0x7f142c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): get_auth_request con 0x7f142c005530 auth_method 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1444a0d640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c005020 0x7f142c0038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1444a0d640 10 monclient(hunting): get_auth_request con 0x7f142c005020 auth_method 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1444a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1444a0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.013+0000 7f1444a0d640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 10 monclient(hunting): handle_auth_done global_id 4774 payload 293 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 -- 10.20.193.51:0/2858302160 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c005020 msgr2=0x7f142c0038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c005020 0x7f142c0038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 -- 10.20.193.51:0/2858302160 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f142c005530 msgr2=0x7f142c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f142c005530 0x7f142c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 10 monclient: _finish_hunting 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 monclient: found mon.c 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 -- 10.20.193.51:0/2858302160 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144015a920 con 0x7f142c009ab0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 10 monclient: _finish_auth 0 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 10 monclient: _check_auth_tickets 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f1446c98640 5 monclient: authenticate success, global_id 4774 2026-02-22T21:39:19.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f1446c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:19.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.014+0000 7f144520e640 1 -- 10.20.193.51:0/2858302160 reap_dead start 2026-02-22T21:39:19.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.015+0000 7f144520e640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 0x7f142c00bea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f143c00a660 tx=0x7f143c02fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f36d12713057e5 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00a2b0 con 0x7f142c009ab0 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: dump: 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f143c035430 con 0x7f142c009ab0 2026-02-22T21:39:19.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 1 -- 10.20.193.51:0/2858302160 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c035640 con 0x7f142c009ab0 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 4 set_mon_vals no callback set 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f14377fe640 10 monclient: dump: 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 10 monclient: shutdown 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 -- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 msgr2=0x7f142c00bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 0x7f142c00bea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f143c00a660 tx=0x7f143c02fa20 comp rx=0 tx=0).stop 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 -- 10.20.193.51:0/2858302160 shutdown_connections 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 --2- 10.20.193.51:0/2858302160 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f142c009ab0 0x7f142c00bea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 -- 10.20.193.51:0/2858302160 >> 10.20.193.51:0/2858302160 conn(0x7f14400cd7b0 msgr2=0x7f14400cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 -- 10.20.193.51:0/2858302160 shutdown_connections 2026-02-22T21:39:19.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 -- 10.20.193.51:0/2858302160 wait complete. 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 10 monclient: build_initial_monmap 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 10 monclient: monmap: 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 librados: starting msgr at 2026-02-22T21:39:19.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.017+0000 7f1446c98640 1 librados: starting objecter 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 Processor -- start 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 -- start start 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 librados: setting wanted keys 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 librados: calling monclient init 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient: init 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c009ab0 0x7f14400736c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-b con 0x7f142c009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-a con 0x7f1440073c00 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f144014c890 0x7f14401f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): picked mon.noname-c con 0x7f144014c890 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1440150600 con 0x7f1440073c00 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f144015bae0 con 0x7f142c009ab0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1440150780 con 0x7f144014c890 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1446c98640 10 monclient(hunting): authenticate will time out at 1234.598565s 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f144520e640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f144014c890 0x7f14401f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1444a0d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c009ab0 0x7f14400736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36920/0 (socket says 10.20.193.51:36920) 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 1 -- 10.20.193.51:0/1058101364 learned_addr learned my addr 10.20.193.51:0/1058101364 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 10 monclient(hunting): get_auth_request con 0x7f1440073c00 auth_method 0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.018+0000 7f1437fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): get_auth_request con 0x7f144014c890 auth_method 0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): get_auth_request con 0x7f142c009ab0 auth_method 0 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1437fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f144520e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.019+0000 7f1444a0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 -- 10.20.193.51:0/1058101364 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f144014c890 msgr2=0x7f14401f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f144014c890 0x7f14401f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 -- 10.20.193.51:0/1058101364 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c009ab0 msgr2=0x7f14400736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c009ab0 0x7f14400736c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 monclient: found mon.noname-a 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.020+0000 7f1437fff640 1 -- 10.20.193.51:0/1058101364 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14400bfa70 con 0x7f1440073c00 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.021+0000 7f1437fff640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f142800ebf0 tx=0x7f142800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.021+0000 7f14357fa640 10 client.?.objecter ms_handle_connect 0x7f1440073c00 2026-02-22T21:39:19.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.021+0000 7f14357fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:19.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142800cad0 con 0x7f1440073c00 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: dump: 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: _finish_auth 0 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: _check_auth_tickets 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142801a070 con 0x7f1440073c00 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 5 monclient: authenticate success, global_id 4812 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 10 monclient: _renew_subs 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14400bfd10 con 0x7f1440073c00 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428015430 con 0x7f1440073c00 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1436ffd640 4 set_mon_vals no callback set 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f14357fa640 10 monclient: dump: 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 10 client.4812.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:19.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 10 monclient: _renew_subs 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14400c02d0 con 0x7f1440073c00 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.023+0000 7f1446c98640 1 librados: init done 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.024+0000 7f1446c98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.024+0000 7f1446c98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.024+0000 7f1446c98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:19.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.024+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14400c73a0 con 0x7f1440073c00 2026-02-22T21:39:19.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.035+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1428014070 con 0x7f1440073c00 2026-02-22T21:39:19.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.035+0000 7f14357fa640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 0x7f141007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:19.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.035+0000 7f1444a0d640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 0x7f141007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:19.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.035+0000 7f1444a0d640 10 monclient: get_auth_request con 0x7f1410078e50 auth_method 0 2026-02-22T21:39:19.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.036+0000 7f14357fa640 10 client.4812.objecter ms_handle_connect 0x7f1410078e50 2026-02-22T21:39:19.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.036+0000 7f1444a0d640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 0x7f141007b310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1430002410 tx=0x7f1430003620 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:19.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.037+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f142805e7c0 con 0x7f1440073c00 2026-02-22T21:39:19.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.037+0000 7f14357fa640 10 client.4812.objecter ms_dispatch2 0x7f1440000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:19.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.037+0000 7f14357fa640 3 client.4812.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:19.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.037+0000 7f14357fa640 3 client.4812.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:19.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.037+0000 7f14357fa640 20 client.4812.objecter dump_active .. 0 homeless 2026-02-22T21:39:19.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.138+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f1428062e80 con 0x7f1440073c00 2026-02-22T21:39:19.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.138+0000 7f14357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:19.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.138+0000 7f14357fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:19.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.238+0000 7f1446c98640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:19.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.238+0000 7f1446c98640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:19.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.238+0000 7f1446c98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:19.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.238+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1440074da0 con 0x7f1440073c00 2026-02-22T21:39:19.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.240+0000 7f14357fa640 1 -- 10.20.193.51:0/1058101364 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f1428069e20 con 0x7f1440073c00 2026-02-22T21:39:19.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.240+0000 7f14357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:19.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.240+0000 7f14357fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:19.242 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:19.242 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:19.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 10 librados: watch_flush enter 2026-02-22T21:39:19.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 10 librados: watch_flush exit 2026-02-22T21:39:19.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 20 client.4812.objecter shutdown clearing up homeless session... 2026-02-22T21:39:19.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 10 client.4812.objecter successfully canceled tick 2026-02-22T21:39:19.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 msgr2=0x7f141007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 0x7f141007b310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1430002410 tx=0x7f1430003620 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 10 monclient: shutdown 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 msgr2=0x7f14401f4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f142800ebf0 tx=0x7f142800c4b0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 shutdown_connections 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1410078e50 0x7f141007b310 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f144014c890 0x7f14401f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1440073c00 0x7f14401f4b50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 --2- 10.20.193.51:0/1058101364 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f142c009ab0 0x7f14400736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 >> 10.20.193.51:0/1058101364 conn(0x7f14400cd7b0 msgr2=0x7f144006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:19.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 shutdown_connections 2026-02-22T21:39:19.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 -- 10.20.193.51:0/1058101364 wait complete. 2026-02-22T21:39:19.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:19.241+0000 7f1446c98640 1 librados: shutdown 2026-02-22T21:39:21.418 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:52.572 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.420 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:52.574 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.429 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:03:52.583 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.446 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:52.562 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.446 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:52.563 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.447 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:52.599 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.449 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:52.601 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.456 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:03:52.572 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:21.459 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:03:52.610 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:39:22.252 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:22.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 10 monclient: get_monmap_and_config 2026-02-22T21:39:22.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 10 monclient: build_initial_monmap 2026-02-22T21:39:22.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 10 monclient: monmap: 2026-02-22T21:39:22.328 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:22.328 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:22.322879+0000 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:22.322879+0000 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:22.329 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 1 Processor -- start 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 1 -- start start 2026-02-22T21:39:22.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.322+0000 7fecb5f6a640 10 monclient: init 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient: _add_conns ranks=[0,4,5,2,3,1] 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecb0150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fecb015c710 0x7fecb00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-b con 0x7fecb015c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fecb0150600 0x7fecb006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-c con 0x7fecb0150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:22.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecb015a920 con 0x7fecb0150220 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fecb015af60 con 0x7fecb015c710 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fecb015b240 con 0x7fecb0150600 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecb5f6a640 10 monclient(hunting): authenticate will time out at 1237.903198s 2026-02-22T21:39:22.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaeffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fecb015c710 0x7fecb00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaf7fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fecb0150220 0x7fecb015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:33938/0 (socket says 10.20.193.51:33938) 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaf7fe640 1 -- 10.20.193.51:0/373819262 learned_addr learned my addr 10.20.193.51:0/373819262 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaffff640 1 --2- 10.20.193.51:0/373819262 >> v2:10.20.193.125:3300/0 conn(0x7fecb0150600 0x7fecb006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaeffd640 10 monclient(hunting): get_auth_request con 0x7fecb015c710 auth_method 0 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaffff640 10 monclient(hunting): get_auth_request con 0x7fecb0150600 auth_method 0 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.323+0000 7fecaffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:22.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.324+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 1 -- 10.20.193.51:0/373819262 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361458201 0 0) 0x7fecb015a920 con 0x7fecb0150220 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 10 monclient(hunting): my global_id is 4815 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecae7fc640 1 -- 10.20.193.51:0/373819262 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec94003620 con 0x7fecb0150220 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 10 monclient(hunting): handle_auth_done global_id 4780 payload 274 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 1 -- 10.20.193.51:0/373819262 >> v2:10.20.193.125:3300/0 conn(0x7fecb0150600 msgr2=0x7fecb006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:22.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 1 --2- 10.20.193.51:0/373819262 >> v2:10.20.193.125:3300/0 conn(0x7fecb0150600 0x7fecb006fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 1 -- 10.20.193.51:0/373819262 >> v1:10.20.193.51:6789/0 conn(0x7fecb0150220 legacy=0x7fecb015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 1 monclient: found mon.noname-b 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.325+0000 7fecaeffd640 1 -- 10.20.193.51:0/373819262 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb015b520 con 0x7fecb015c710 2026-02-22T21:39:22.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.326+0000 7fecaeffd640 1 --2- 10.20.193.51:0/373819262 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb015c710 0x7fecb00c73a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7feca00098a0 tx=0x7feca002eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f1f42991e38256 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:22.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 1 -- 10.20.193.51:0/373819262 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca003c070 con 0x7fecb015c710 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: dump: 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:22.343 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 1 -- 10.20.193.51:0/373819262 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca002fb80 con 0x7fecb015c710 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 1 -- 10.20.193.51:0/373819262 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0037440 con 0x7fecb015c710 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 5 monclient: authenticate success, global_id 4780 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 4 set_mon_vals no callback set 2026-02-22T21:39:22.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecae7fc640 10 monclient: dump: 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 10 monclient: shutdown 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 -- 10.20.193.51:0/373819262 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb015c710 msgr2=0x7fecb00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/373819262 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb015c710 0x7fecb00c73a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7feca00098a0 tx=0x7feca002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 -- 10.20.193.51:0/373819262 shutdown_connections 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/373819262 >> v2:10.20.193.125:3300/0 conn(0x7fecb0150600 0x7fecb006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/373819262 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb015c710 0x7fecb00c73a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 -- 10.20.193.51:0/373819262 >> 10.20.193.51:0/373819262 conn(0x7fecb00cd7b0 msgr2=0x7fecb00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 -- 10.20.193.51:0/373819262 shutdown_connections 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 1 -- 10.20.193.51:0/373819262 wait complete. 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 10 monclient: build_initial_monmap 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.328+0000 7fecb5f6a640 10 monclient: monmap: 2026-02-22T21:39:22.346 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 librados: starting msgr at 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 librados: starting objecter 2026-02-22T21:39:22.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 Processor -- start 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 -- start start 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 librados: setting wanted keys 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 librados: calling monclient init 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient: init 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fecb0150220 0x7fecb01f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-a con 0x7fecb0150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-b con 0x7fecb0150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fecb015c710 0x7fecb0073970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): picked mon.noname-c con 0x7fecb015c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:22.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fecb00701e0 con 0x7fecb0150220 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fecb015bae0 con 0x7fecb0150600 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fecb0070360 con 0x7fecb015c710 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecb5f6a640 10 monclient(hunting): authenticate will time out at 1237.909653s 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.329+0000 7fecaf7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fecb0150220 0x7fecb01f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:41746/0 (socket says 10.20.193.51:41746) 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 1 -- 10.20.193.51:0/751976457 learned_addr learned my addr 10.20.193.51:0/751976457 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): get_auth_request con 0x7fecb0150600 auth_method 0 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fecb015c710 0x7fecb0073970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): get_auth_request con 0x7fecb0150220 auth_method 0 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): get_auth_request con 0x7fecb015c710 auth_method 0 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.330+0000 7fecaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:22.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 10 monclient(hunting): handle_auth_done global_id 4783 payload 824 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 -- 10.20.193.51:0/751976457 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fecb015c710 msgr2=0x7fecb0073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fecb015c710 0x7fecb0073970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 -- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fecb0150220 msgr2=0x7fecb01f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fecb0150220 0x7fecb01f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 monclient: found mon.noname-b 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.331+0000 7fecaeffd640 1 -- 10.20.193.51:0/751976457 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb0155fa0 con 0x7fecb0150600 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.332+0000 7fecaeffd640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feca003a040 tx=0x7feca00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:22.352 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.332+0000 7fec8bfff640 10 client.?.objecter ms_handle_connect 0x7fecb0150600 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.332+0000 7fec8bfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0046020 con 0x7fecb0150600 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: dump: 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:22.353 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: _finish_auth 0 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca0042430 con 0x7fecb0150600 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 5 monclient: authenticate success, global_id 4783 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 10 monclient: _renew_subs 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb0156280 con 0x7fecb0150600 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca003c040 con 0x7fecb0150600 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:22.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fec8bfff640 10 monclient: dump: 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecadffb640 4 set_mon_vals no callback set 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 10 client.4783.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 10 monclient: _renew_subs 2026-02-22T21:39:22.355 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb0073170 con 0x7fecb0150600 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.336+0000 7fecb5f6a640 1 librados: init done 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.337+0000 7fecb5f6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.337+0000 7fecb5f6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.337+0000 7fecb5f6a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.337+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb006fca0 con 0x7fecb0150600 2026-02-22T21:39:22.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.349+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7feca0053070 con 0x7fecb0150600 2026-02-22T21:39:22.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.350+0000 7fec8bfff640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 0x7fec8007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:22.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.350+0000 7fecaf7fe640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 0x7fec8007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:22.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.350+0000 7fecaf7fe640 10 monclient: get_auth_request con 0x7fec80078e50 auth_method 0 2026-02-22T21:39:22.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.350+0000 7fecaf7fe640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 0x7fec8007b310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fec9c007b60 tx=0x7fec9c007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.350+0000 7fec8bfff640 10 client.4783.objecter ms_handle_connect 0x7fec80078e50 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.351+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7feca0083430 con 0x7fecb0150600 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.351+0000 7fec8bfff640 10 client.4783.objecter ms_dispatch2 0x7fecb0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.351+0000 7fec8bfff640 3 client.4783.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.351+0000 7fec8bfff640 3 client.4783.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:22.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.351+0000 7fec8bfff640 20 client.4783.objecter dump_active .. 0 homeless 2026-02-22T21:39:22.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.453+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7feca0050020 con 0x7fecb0150600 2026-02-22T21:39:22.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.453+0000 7fec8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:22.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.453+0000 7fec8bfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:22.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.556+0000 7fecb5f6a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:22.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.556+0000 7fecb5f6a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:22.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.556+0000 7fecb5f6a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:22.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.556+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fecb00c4270 con 0x7fecb0150600 2026-02-22T21:39:22.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.558+0000 7fec8bfff640 1 -- 10.20.193.51:0/751976457 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7feca00873f0 con 0x7fecb0150600 2026-02-22T21:39:22.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.558+0000 7fec8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:22.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.558+0000 7fec8bfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:22.561 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:22.561 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:22.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 10 librados: watch_flush enter 2026-02-22T21:39:22.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 10 librados: watch_flush exit 2026-02-22T21:39:22.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 20 client.4783.objecter shutdown clearing up homeless session... 2026-02-22T21:39:22.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 10 client.4783.objecter successfully canceled tick 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 msgr2=0x7fec8007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 0x7fec8007b310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fec9c007b60 tx=0x7fec9c007ab0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 10 monclient: shutdown 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 msgr2=0x7fecb014c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:22.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feca003a040 tx=0x7feca00047c0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 shutdown_connections 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fec80078e50 0x7fec8007b310 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fecb015c710 0x7fecb0073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fecb0150600 0x7fecb014c800 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 --2- 10.20.193.51:0/751976457 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fecb0150220 0x7fecb01f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:22.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 >> 10.20.193.51:0/751976457 conn(0x7fecb00cd7b0 msgr2=0x7fecb00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:22.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 shutdown_connections 2026-02-22T21:39:22.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 -- 10.20.193.51:0/751976457 wait complete. 2026-02-22T21:39:22.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:22.560+0000 7fecb5f6a640 1 librados: shutdown 2026-02-22T21:39:25.572 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:25.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.638+0000 7fc58c1e3640 10 monclient: get_monmap_and_config 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.638+0000 7fc58c1e3640 10 monclient: build_initial_monmap 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.638+0000 7fc58c1e3640 10 monclient: monmap: 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:25.639482+0000 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:25.639482+0000 2026-02-22T21:39:25.641 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 Processor -- start 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 -- start start 2026-02-22T21:39:25.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient: init 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient: _add_conns ranks=[3,2,1,5,0,4] 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc58415f710 0x7fc58415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-a con 0x7fc58415f710 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5840700e0 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fc58415dab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5840caca0 con 0x7fc58415dab0 2026-02-22T21:39:25.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5840caf80 con 0x7fc5840700e0 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc5840cbb90 con 0x7fc58415f710 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.639+0000 7fc58c1e3640 10 monclient(hunting): authenticate will time out at 1241.219908s 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc58415f710 0x7fc58415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc58415f710 0x7fc58415faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36946/0 (socket says 10.20.193.51:36946) 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 1 -- 10.20.193.51:0/1688111757 learned_addr learned my addr 10.20.193.51:0/1688111757 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): get_auth_request con 0x7fc58415f710 auth_method 0 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:25.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc58a759640 1 --1- 10.20.193.51:0/1688111757 >> v1:10.20.193.68:6789/0 conn(0x7fc58415dab0 0x7fc5840ca040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:42742/0 (socket says 10.20.193.51:42742) 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.640+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 10 monclient(hunting): handle_auth_done global_id 4821 payload 274 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 1 -- 10.20.193.51:0/1688111757 >> v1:10.20.193.125:6789/0 conn(0x7fc5840700e0 legacy=0x7fc58415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 1 -- 10.20.193.51:0/1688111757 >> v1:10.20.193.68:6789/0 conn(0x7fc58415dab0 legacy=0x7fc5840ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 10 monclient: _finish_hunting 0 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 1 monclient: found mon.noname-a 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.641+0000 7fc589f58640 1 -- 10.20.193.51:0/1688111757 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5840cb260 con 0x7fc58415f710 2026-02-22T21:39:25.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.642+0000 7fc589f58640 1 --2- 10.20.193.51:0/1688111757 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc58415f710 0x7fc58415faf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc574009970 tx=0x7fc57402ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e70910e26d0e8e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:25.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 1 -- 10.20.193.51:0/1688111757 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57403c070 con 0x7fc58415f710 2026-02-22T21:39:25.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:25.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: dump: 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:25.655 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: _finish_auth 0 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: _check_auth_tickets 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 1 -- 10.20.193.51:0/1688111757 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc57402fc50 con 0x7fc58415f710 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 5 monclient: authenticate success, global_id 4821 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 1 -- 10.20.193.51:0/1688111757 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574037460 con 0x7fc58415f710 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 4 set_mon_vals no callback set 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:25.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc588f56640 10 monclient: dump: 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 10 monclient: shutdown 2026-02-22T21:39:25.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1688111757 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc58415f710 msgr2=0x7fc58415faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1688111757 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc58415f710 0x7fc58415faf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc574009970 tx=0x7fc57402ef40 comp rx=0 tx=0).stop 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1688111757 shutdown_connections 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1688111757 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc58415f710 0x7fc58415faf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1688111757 >> 10.20.193.51:0/1688111757 conn(0x7fc5840cf6c0 msgr2=0x7fc5840cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1688111757 shutdown_connections 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.644+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1688111757 wait complete. 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 10 monclient: build_initial_monmap 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 10 monclient: monmap: 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:25.658 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 librados: starting msgr at 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 librados: starting objecter 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 Processor -- start 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 -- start start 2026-02-22T21:39:25.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 librados: setting wanted keys 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 1 librados: calling monclient init 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.645+0000 7fc58c1e3640 10 monclient: init 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5840700e0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc58415dab0 0x7fc5841f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-b con 0x7fc58415dab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc58415f710 0x7fc58406cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): picked mon.noname-c con 0x7fc58415f710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc5840ca750 con 0x7fc5840700e0 2026-02-22T21:39:25.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc5840cb820 con 0x7fc58415dab0 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc5840ca8d0 con 0x7fc58415f710 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58c1e3640 10 monclient(hunting): authenticate will time out at 1241.226286s 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589757640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc58415dab0 0x7fc5841f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36952/0 (socket says 10.20.193.51:36952) 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 1 -- 10.20.193.51:0/1155341903 learned_addr learned my addr 10.20.193.51:0/1155341903 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): get_auth_request con 0x7fc5840700e0 auth_method 0 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:25.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58a759640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc58415f710 0x7fc58406cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589757640 10 monclient(hunting): get_auth_request con 0x7fc58415dab0 auth_method 0 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589757640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589757640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589757640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc589f58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58a759640 10 monclient(hunting): get_auth_request con 0x7fc58415f710 auth_method 0 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58a759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58a759640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:25.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.646+0000 7fc58a759640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc589757640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc589757640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc589757640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc58a759640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc58a759640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.647+0000 7fc58a759640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 10 monclient(hunting): handle_auth_done global_id 4824 payload 824 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 -- 10.20.193.51:0/1155341903 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc58415f710 msgr2=0x7fc58406cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc58415f710 0x7fc58406cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 -- 10.20.193.51:0/1155341903 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc58415dab0 msgr2=0x7fc5841f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc58415dab0 0x7fc5841f7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 10 monclient: _finish_hunting 0 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 monclient: found mon.noname-a 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 -- 10.20.193.51:0/1155341903 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5841f4080 con 0x7fc5840700e0 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.648+0000 7fc589f58640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc57402f470 tx=0x7fc5740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.649+0000 7fc5727fc640 10 client.?.objecter ms_handle_connect 0x7fc5840700e0 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.649+0000 7fc5727fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:25.664 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574047070 con 0x7fc5840700e0 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: dump: 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:25.665 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc574042430 con 0x7fc5840700e0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 5 monclient: authenticate success, global_id 4824 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 10 monclient: _renew_subs 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5841f4360 con 0x7fc5840700e0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57403c040 con 0x7fc5840700e0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:25.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc5727fc640 10 monclient: dump: 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:25.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc573fff640 4 set_mon_vals no callback set 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 10 client.4824.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 10 monclient: _renew_subs 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5841f4920 con 0x7fc5840700e0 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.650+0000 7fc58c1e3640 1 librados: init done 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.651+0000 7fc58c1e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.651+0000 7fc58c1e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.651+0000 7fc58c1e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.651+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58415faf0 con 0x7fc5840700e0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.662+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc574042800 con 0x7fc5840700e0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.662+0000 7fc5727fc640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 0x7fc55407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.663+0000 7fc589757640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 0x7fc55407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.663+0000 7fc589757640 10 monclient: get_auth_request con 0x7fc554078e50 auth_method 0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.663+0000 7fc5727fc640 10 client.4824.objecter ms_handle_connect 0x7fc554078e50 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.663+0000 7fc589757640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 0x7fc55407b310 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc58414cea0 tx=0x7fc578007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.664+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc5740825b0 con 0x7fc5840700e0 2026-02-22T21:39:25.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.664+0000 7fc5727fc640 10 client.4824.objecter ms_dispatch2 0x7fc584000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:25.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.664+0000 7fc5727fc640 3 client.4824.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:25.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.664+0000 7fc5727fc640 3 client.4824.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:25.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.664+0000 7fc5727fc640 20 client.4824.objecter dump_active .. 0 homeless 2026-02-22T21:39:25.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.765+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc574086570 con 0x7fc5840700e0 2026-02-22T21:39:25.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.765+0000 7fc5727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:25.767 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.765+0000 7fc5727fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:25.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.864+0000 7fc58c1e3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:25.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.864+0000 7fc58c1e3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:25.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.864+0000 7fc58c1e3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:25.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.864+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc58415f280 con 0x7fc5840700e0 2026-02-22T21:39:25.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.869+0000 7fc5727fc640 1 -- 10.20.193.51:0/1155341903 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fc57408a870 con 0x7fc5840700e0 2026-02-22T21:39:25.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.869+0000 7fc5727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:25.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.869+0000 7fc5727fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:25.871 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:25.871 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:25.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.870+0000 7fc58c1e3640 10 librados: watch_flush enter 2026-02-22T21:39:25.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.870+0000 7fc58c1e3640 10 librados: watch_flush exit 2026-02-22T21:39:25.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.870+0000 7fc58c1e3640 20 client.4824.objecter shutdown clearing up homeless session... 2026-02-22T21:39:25.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.870+0000 7fc58c1e3640 10 client.4824.objecter successfully canceled tick 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 msgr2=0x7fc55407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 0x7fc55407b310 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc58414cea0 tx=0x7fc578007f30 comp rx=0 tx=0).stop 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 10 monclient: shutdown 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 msgr2=0x7fc5840cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc57402f470 tx=0x7fc5740047c0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 shutdown_connections 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc554078e50 0x7fc55407b310 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc58415f710 0x7fc58406cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc58415dab0 0x7fc5841f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 --2- 10.20.193.51:0/1155341903 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc5840700e0 0x7fc5840cefa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:25.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 >> 10.20.193.51:0/1155341903 conn(0x7fc5840cf6c0 msgr2=0x7fc58415d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:25.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 shutdown_connections 2026-02-22T21:39:25.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 -- 10.20.193.51:0/1155341903 wait complete. 2026-02-22T21:39:25.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:25.871+0000 7fc58c1e3640 1 librados: shutdown 2026-02-22T21:39:28.882 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:28.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.949+0000 7fdbc54c9640 10 monclient: get_monmap_and_config 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.949+0000 7fdbc54c9640 10 monclient: build_initial_monmap 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.949+0000 7fdbc54c9640 10 monclient: monmap: 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:28.950570+0000 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:28.950570+0000 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:28.952 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 Processor -- start 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 -- start start 2026-02-22T21:39:28.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient: init 2026-02-22T21:39:28.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient: _add_conns ranks=[0,5,2,4,1,3] 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbc0150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdbc015c710 0x7fdbc00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbc0150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc015a920 con 0x7fdbc0150220 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc015af60 con 0x7fdbc0150600 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fdbc015b240 con 0x7fdbc015c710 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:28.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.950+0000 7fdbc54c9640 10 monclient(hunting): authenticate will time out at 1244.530992s 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.951+0000 7fdbbeffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fdbc0150220 0x7fdbc015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:52324/0 (socket says 10.20.193.51:52324) 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.951+0000 7fdbbeffd640 1 -- 10.20.193.51:0/2732846523 learned_addr learned my addr 10.20.193.51:0/2732846523 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbe7fc640 1 --2- 10.20.193.51:0/2732846523 >> v2:10.20.193.125:3300/0 conn(0x7fdbc015c710 0x7fdbc00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205408541 0 0) 0x7fdbc015a920 con 0x7fdbc0150220 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 10 monclient(hunting): _negotiate 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 10 monclient(hunting): my global_id is 4827 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.952+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbac003620 con 0x7fdbc0150220 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): get_auth_request con 0x7fdbc015c710 auth_method 0 2026-02-22T21:39:28.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216757694 0 0) 0x7fdbc015af60 con 0x7fdbc0150600 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 10 monclient(hunting): _negotiate 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 10 monclient(hunting): my global_id is 4751 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc015a920 con 0x7fdbc0150600 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:28.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.953+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_done global_id 4754 payload 274 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/2732846523 >> v1:10.20.193.125:6789/0 conn(0x7fdbc0150600 legacy=0x7fdbc006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/2732846523 >> v1:10.20.193.51:6789/0 conn(0x7fdbc0150220 legacy=0x7fdbc015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 1 monclient: found mon.noname-c 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:28.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.954+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/2732846523 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc015b520 con 0x7fdbc015c710 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.956+0000 7fdbbe7fc640 1 --2- 10.20.193.51:0/2732846523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc015c710 0x7fdbc00c73a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098d0 tx=0x7fdba802ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d91b4c9fba1d7570 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba803c070 con 0x7fdbc015c710 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: dump: 2026-02-22T21:39:28.967 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: _finish_auth 0 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: _check_auth_tickets 2026-02-22T21:39:28.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba802fbf0 con 0x7fdbc015c710 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 1 -- 10.20.193.51:0/2732846523 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8037440 con 0x7fdbc015c710 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbbdffb640 10 monclient: dump: 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:28.969 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 5 monclient: authenticate success, global_id 4754 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 4 set_mon_vals no callback set 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 10 monclient: shutdown 2026-02-22T21:39:28.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 1 -- 10.20.193.51:0/2732846523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc015c710 msgr2=0x7fdbc00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/2732846523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc015c710 0x7fdbc00c73a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098d0 tx=0x7fdba802ef20 comp rx=0 tx=0).stop 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 1 -- 10.20.193.51:0/2732846523 shutdown_connections 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/2732846523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc015c710 0x7fdbc00c73a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.958+0000 7fdbc54c9640 1 -- 10.20.193.51:0/2732846523 >> 10.20.193.51:0/2732846523 conn(0x7fdbc00cd7b0 msgr2=0x7fdbc00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 -- 10.20.193.51:0/2732846523 shutdown_connections 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 -- 10.20.193.51:0/2732846523 wait complete. 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 10 monclient: build_initial_monmap 2026-02-22T21:39:28.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 10 monclient: monmap: 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 librados: starting msgr at 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 librados: starting objecter 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 Processor -- start 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 -- start start 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 librados: setting wanted keys 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 1 librados: calling monclient init 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.959+0000 7fdbc54c9640 10 monclient: init 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc0150220 0x7fdbc00cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:28.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdbc015c710 0x7fdbc01f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc00703b0 con 0x7fdbc0150600 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc015bae0 con 0x7fdbc015c710 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc0070530 con 0x7fdbc0150220 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:28.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbc54c9640 10 monclient(hunting): authenticate will time out at 1244.540231s 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbeffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc0150220 0x7fdbc00cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbf7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdbc015c710 0x7fdbc01f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:58976/0 (socket says 10.20.193.51:58976) 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/1290189954 learned_addr learned my addr 10.20.193.51:0/1290189954 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): get_auth_request con 0x7fdbc0150600 auth_method 0 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbeffd640 10 monclient(hunting): get_auth_request con 0x7fdbc0150220 auth_method 0 2026-02-22T21:39:28.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbf7fe640 10 monclient(hunting): get_auth_request con 0x7fdbc015c710 auth_method 0 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.960+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:28.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.961+0000 7fdbbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 10 monclient(hunting): handle_auth_done global_id 4830 payload 824 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/1290189954 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc0150220 msgr2=0x7fdbc00cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc0150220 0x7fdbc00cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/1290189954 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdbc015c710 msgr2=0x7fdbc01f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdbc015c710 0x7fdbc01f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 monclient: found mon.noname-a 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:28.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.962+0000 7fdbbe7fc640 1 -- 10.20.193.51:0/1290189954 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc01e6db0 con 0x7fdbc0150600 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.963+0000 7fdbbe7fc640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098d0 tx=0x7fdba80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.963+0000 7fdb9f7fe640 10 client.?.objecter ms_handle_connect 0x7fdbc0150600 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.963+0000 7fdb9f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8046020 con 0x7fdbc0150600 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: dump: 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.978 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba8042650 con 0x7fdbc0150600 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba803c040 con 0x7fdbc0150600 2026-02-22T21:39:28.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdb9f7fe640 10 monclient: dump: 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:28.980 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdbbd7fa640 4 set_mon_vals no callback set 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdbc54c9640 5 monclient: authenticate success, global_id 4830 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.964+0000 7fdbc54c9640 10 monclient: _renew_subs 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc01e7090 con 0x7fdbc0150600 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 client.4830.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 monclient: _renew_subs 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc01e7650 con 0x7fdbc0150600 2026-02-22T21:39:28.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 1 librados: init done 2026-02-22T21:39:28.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:28.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:28.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:28.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:28.965+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc00c73a0 con 0x7fdbc0150600 2026-02-22T21:39:29.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.072+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fdba804a3b0 con 0x7fdbc0150600 2026-02-22T21:39:29.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.073+0000 7fdb9f7fe640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 0x7fdb8c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:29.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.073+0000 7fdbbeffd640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 0x7fdb8c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:29.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.073+0000 7fdbbeffd640 10 monclient: get_auth_request con 0x7fdb8c078e00 auth_method 0 2026-02-22T21:39:29.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.073+0000 7fdbbeffd640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 0x7fdb8c07b2c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4005980 tx=0x7fdbb4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:29.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.073+0000 7fdb9f7fe640 10 client.4830.objecter ms_handle_connect 0x7fdb8c078e00 2026-02-22T21:39:29.083 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fdba802fab0 con 0x7fdbc0150600 2026-02-22T21:39:29.083 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 10 client.4830.objecter ms_dispatch2 0x7fdbc0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:29.083 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 3 client.4830.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:29.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 3 client.4830.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:29.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 20 client.4830.objecter dump_active .. 0 homeless 2026-02-22T21:39:29.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fdba8050020 con 0x7fdbc0150600 2026-02-22T21:39:29.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:29.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.081+0000 7fdb9f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:29.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.181+0000 7fdbc54c9640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:29.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.181+0000 7fdbc54c9640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:29.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.181+0000 7fdbc54c9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:29.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.181+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdbc00c4270 con 0x7fdbc0150600 2026-02-22T21:39:29.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.184+0000 7fdb9f7fe640 1 -- 10.20.193.51:0/1290189954 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fdba80869a0 con 0x7fdbc0150600 2026-02-22T21:39:29.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.184+0000 7fdb9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:29.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.184+0000 7fdb9f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:29.185 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:29.185 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 10 librados: watch_flush enter 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 10 librados: watch_flush exit 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 20 client.4830.objecter shutdown clearing up homeless session... 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 10 client.4830.objecter successfully canceled tick 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 msgr2=0x7fdb8c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 0x7fdb8c07b2c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4005980 tx=0x7fdbb4009040 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 10 monclient: shutdown 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 msgr2=0x7fdbc006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098d0 tx=0x7fdba80047c0 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 shutdown_connections 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdb8c078e00 0x7fdb8c07b2c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdbc015c710 0x7fdbc01f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdbc0150600 0x7fdbc006e0e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 --2- 10.20.193.51:0/1290189954 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdbc0150220 0x7fdbc00cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 >> 10.20.193.51:0/1290189954 conn(0x7fdbc00cd7b0 msgr2=0x7fdbc00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 shutdown_connections 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 -- 10.20.193.51:0/1290189954 wait complete. 2026-02-22T21:39:29.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:29.185+0000 7fdbc54c9640 1 librados: shutdown 2026-02-22T21:39:32.196 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:32.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 10 monclient: get_monmap_and_config 2026-02-22T21:39:32.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 10 monclient: build_initial_monmap 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 10 monclient: monmap: 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:32.265111+0000 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:32.265111+0000 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 1 Processor -- start 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 1 -- start start 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.264+0000 7f9762216640 10 monclient: init 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient: _add_conns ranks=[4,2,5,1,3,0] 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f975c150220 0x7f975c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): picked mon.noname-b con 0x7f975c150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f975c150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f975c15c710 0x7f975c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): picked mon.noname-c con 0x7f975c15c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f975c15a920 con 0x7f975c150600 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f975c15af60 con 0x7f975c150220 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f975c15b240 con 0x7f975c15c710 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f9762216640 10 monclient(hunting): authenticate will time out at 1247.845415s 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f975b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f975c150220 0x7f975c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.265+0000 7f975bfff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f975c15c710 0x7f975c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f975c150220 0x7f975c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:37502/0 (socket says 10.20.193.51:37502) 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 1 -- 10.20.193.51:0/2458561784 learned_addr learned my addr 10.20.193.51:0/2458561784 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): get_auth_request con 0x7f975c150220 auth_method 0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): get_auth_request con 0x7f975c15c710 auth_method 0 2026-02-22T21:39:32.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.266+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 10 monclient(hunting): handle_auth_done global_id 4795 payload 274 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 1 -- 10.20.193.51:0/2458561784 >> v2:10.20.193.125:3300/0 conn(0x7f975c15c710 msgr2=0x7f975c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 1 --2- 10.20.193.51:0/2458561784 >> v2:10.20.193.125:3300/0 conn(0x7f975c15c710 0x7f975c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 1 -- 10.20.193.51:0/2458561784 >> v1:10.20.193.125:6789/0 conn(0x7f975c150600 legacy=0x7f975c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 1 monclient: found mon.noname-b 2026-02-22T21:39:32.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:32.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:32.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.267+0000 7f975b7fe640 1 -- 10.20.193.51:0/2458561784 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c15b520 con 0x7f975c150220 2026-02-22T21:39:32.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.268+0000 7f975b7fe640 1 --2- 10.20.193.51:0/2458561784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c0c73a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f975c0c65c0 tx=0x7f974802ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec81041e6272c5d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 1 -- 10.20.193.51:0/2458561784 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974803c070 con 0x7f975c150220 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: dump: 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 1 -- 10.20.193.51:0/2458561784 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974802fc70 con 0x7f975c150220 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 5 monclient: authenticate success, global_id 4795 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 1 -- 10.20.193.51:0/2458561784 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748037660 con 0x7f975c150220 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f975a7fc640 10 monclient: dump: 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 4 set_mon_vals no callback set 2026-02-22T21:39:32.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 10 monclient: shutdown 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 -- 10.20.193.51:0/2458561784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 msgr2=0x7f975c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 --2- 10.20.193.51:0/2458561784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c0c73a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f975c0c65c0 tx=0x7f974802ef60 comp rx=0 tx=0).stop 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 -- 10.20.193.51:0/2458561784 shutdown_connections 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 --2- 10.20.193.51:0/2458561784 >> v2:10.20.193.125:3300/0 conn(0x7f975c15c710 0x7f975c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 --2- 10.20.193.51:0/2458561784 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c0c73a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 -- 10.20.193.51:0/2458561784 >> 10.20.193.51:0/2458561784 conn(0x7f975c0cd7b0 msgr2=0x7f975c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 -- 10.20.193.51:0/2458561784 shutdown_connections 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.270+0000 7f9762216640 1 -- 10.20.193.51:0/2458561784 wait complete. 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 10 monclient: build_initial_monmap 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 10 monclient: monmap: 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:32.273 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 librados: starting msgr at 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 librados: starting objecter 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 Processor -- start 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 -- start start 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 librados: setting wanted keys 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 1 librados: calling monclient init 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.271+0000 7f9762216640 10 monclient: init 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c1f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): picked mon.noname-b con 0x7f975c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f975c150600 0x7f975c0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): picked mon.noname-a con 0x7f975c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): picked mon.noname-c con 0x7f975c15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0701e0 con 0x7f975c150600 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f975c15bae0 con 0x7f975c150220 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f975c070360 con 0x7f975c15c710 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f9762216640 10 monclient(hunting): authenticate will time out at 1247.852061s 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975affd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f975c150600 0x7f975c0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975b7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c1f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:33058/0 (socket says 10.20.193.51:33058) 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 1 -- 10.20.193.51:0/4012469746 learned_addr learned my addr 10.20.193.51:0/4012469746 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): get_auth_request con 0x7f975c15c710 auth_method 0 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:32.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975affd640 10 monclient(hunting): get_auth_request con 0x7f975c150600 auth_method 0 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975b7fe640 10 monclient(hunting): get_auth_request con 0x7f975c150220 auth_method 0 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.272+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.273+0000 7f975b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 10 monclient(hunting): handle_auth_done global_id 4766 payload 824 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 -- 10.20.193.51:0/4012469746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 msgr2=0x7f975c1f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c1f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 -- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f975c150600 msgr2=0x7f975c0c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f975c150600 0x7f975c0c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 monclient: found mon.noname-c 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.274+0000 7f975bfff640 1 -- 10.20.193.51:0/4012469746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c1f2760 con 0x7f975c15c710 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.275+0000 7f975bfff640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f975000efc0 tx=0x7f975000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.275+0000 7f9737fff640 10 client.?.objecter ms_handle_connect 0x7f975c15c710 2026-02-22T21:39:32.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.275+0000 7f9737fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750018020 con 0x7f975c15c710 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: dump: 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: _finish_auth 0 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: _check_auth_tickets 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 5 monclient: authenticate success, global_id 4766 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9750014480 con 0x7f975c15c710 2026-02-22T21:39:32.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750014870 con 0x7f975c15c710 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _renew_subs 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c0c1ad0 con 0x7f975c15c710 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9759ffb640 4 set_mon_vals no callback set 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9737fff640 10 monclient: dump: 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 client.4766.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _renew_subs 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c0c2090 con 0x7f975c15c710 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 1 librados: init done 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:32.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:32.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.277+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f975c06fca0 con 0x7f975c15c710 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.288+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9750014a10 con 0x7f975c15c710 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.289+0000 7f9737fff640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 0x7f972c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.289+0000 7f975b7fe640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 0x7f972c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.289+0000 7f975b7fe640 10 monclient: get_auth_request con 0x7f972c078e50 auth_method 0 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.289+0000 7f975b7fe640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 0x7f972c07b310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f975c074940 tx=0x7f9748039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:32.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.289+0000 7f9737fff640 10 client.4766.objecter ms_handle_connect 0x7f972c078e50 2026-02-22T21:39:32.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.290+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f975005ee80 con 0x7f975c15c710 2026-02-22T21:39:32.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.290+0000 7f9737fff640 10 client.4766.objecter ms_dispatch2 0x7f975c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:32.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.290+0000 7f9737fff640 3 client.4766.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:32.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.290+0000 7f9737fff640 3 client.4766.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:32.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.290+0000 7f9737fff640 20 client.4766.objecter dump_active .. 0 homeless 2026-02-22T21:39:32.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.393+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9750022030 con 0x7f975c15c710 2026-02-22T21:39:32.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.393+0000 7f9737fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:32.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.393+0000 7f9737fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:32.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.493+0000 7f9762216640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:32.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.493+0000 7f9762216640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:32.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.493+0000 7f9762216640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:32.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.493+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f975c157f60 con 0x7f975c15c710 2026-02-22T21:39:32.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.499+0000 7f9737fff640 1 -- 10.20.193.51:0/4012469746 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f975c1f2760 con 0x7f975c15c710 2026-02-22T21:39:32.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.499+0000 7f9737fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:32.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.499+0000 7f9737fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:32.501 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:32.501 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 10 librados: watch_flush enter 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 10 librados: watch_flush exit 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 20 client.4766.objecter shutdown clearing up homeless session... 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 10 client.4766.objecter successfully canceled tick 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 msgr2=0x7f972c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 0x7f972c07b310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f975c074940 tx=0x7f9748039040 comp rx=0 tx=0).stop 2026-02-22T21:39:32.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 10 monclient: shutdown 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 msgr2=0x7f975c1e6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f975000efc0 tx=0x7f975000c490 comp rx=0 tx=0).stop 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 shutdown_connections 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f972c078e50 0x7f972c07b310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f975c15c710 0x7f975c1e6d80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f975c150600 0x7f975c0c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 --2- 10.20.193.51:0/4012469746 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f975c150220 0x7f975c1f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 >> 10.20.193.51:0/4012469746 conn(0x7f975c0cd7b0 msgr2=0x7f975c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 shutdown_connections 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 -- 10.20.193.51:0/4012469746 wait complete. 2026-02-22T21:39:32.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:32.500+0000 7f9762216640 1 librados: shutdown 2026-02-22T21:39:35.510 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.579+0000 7f8bb6ec9640 10 monclient: get_monmap_and_config 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.579+0000 7f8bb6ec9640 10 monclient: build_initial_monmap 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.579+0000 7f8bb6ec9640 10 monclient: monmap: 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:35.580538+0000 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:35.580538+0000 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:35.581 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 Processor -- start 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 -- start start 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient: init 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient: _add_conns ranks=[2,0,5,1,3,4] 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bb0150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bb015c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f8bb00703b0 0x7f8bb00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb00703b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:35.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb015a920 con 0x7f8bb015c710 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb015af60 con 0x7f8bb0150220 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb015b240 con 0x7f8bb00703b0 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.580+0000 7f8bb6ec9640 10 monclient(hunting): authenticate will time out at 1251.160839s 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8ba7fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f8bb015c710 0x7f8bb006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:52330/0 (socket says 10.20.193.51:52330) 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8ba7fff640 1 -- 10.20.193.51:0/419365600 learned_addr learned my addr 10.20.193.51:0/419365600 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8bb543f640 1 --2- 10.20.193.51:0/419365600 >> v2:10.20.193.125:3300/0 conn(0x7f8bb00703b0 0x7f8bb00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8bb543f640 10 monclient(hunting): get_auth_request con 0x7f8bb00703b0 auth_method 0 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8bb543f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8bb543f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.581+0000 7f8bb543f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:35.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 1 -- 10.20.193.51:0/419365600 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233996463 0 0) 0x7f8bb015a920 con 0x7f8bb015c710 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 10 monclient(hunting): _negotiate 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 10 monclient(hunting): my global_id is 4836 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.582+0000 7f8ba77fe640 1 -- 10.20.193.51:0/419365600 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b9c003620 con 0x7f8bb015c710 2026-02-22T21:39:35.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 1 -- 10.20.193.51:0/419365600 >> v1:10.20.193.125:6789/0 conn(0x7f8bb0150220 legacy=0x7f8bb015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 1 -- 10.20.193.51:0/419365600 >> v1:10.20.193.51:6789/0 conn(0x7f8bb015c710 legacy=0x7f8bb006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 10 monclient: _finish_hunting 0 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 1 monclient: found mon.noname-c 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:35.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.583+0000 7f8bb543f640 1 -- 10.20.193.51:0/419365600 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb015b520 con 0x7f8bb00703b0 2026-02-22T21:39:35.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.586+0000 7f8bb543f640 1 --2- 10.20.193.51:0/419365600 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb00703b0 0x7f8bb00c73a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009dc0 tx=0x7f8bac02f2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762ad52493750b81 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 1 -- 10.20.193.51:0/419365600 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac03c070 con 0x7f8bb00703b0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: dump: 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: _finish_auth 0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: _check_auth_tickets 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 1 -- 10.20.193.51:0/419365600 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac037440 con 0x7f8bb00703b0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 5 monclient: authenticate success, global_id 4769 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 1 -- 10.20.193.51:0/419365600 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac037810 con 0x7f8bb00703b0 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 4 set_mon_vals no callback set 2026-02-22T21:39:35.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8ba77fe640 10 monclient: dump: 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 10 monclient: shutdown 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/419365600 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb00703b0 msgr2=0x7f8bb00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/419365600 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb00703b0 0x7f8bb00c73a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009dc0 tx=0x7f8bac02f2f0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/419365600 shutdown_connections 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/419365600 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb00703b0 0x7f8bb00c73a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/419365600 >> 10.20.193.51:0/419365600 conn(0x7f8bb00cd7b0 msgr2=0x7f8bb00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/419365600 shutdown_connections 2026-02-22T21:39:35.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.588+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/419365600 wait complete. 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient: build_initial_monmap 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient: monmap: 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 librados: starting msgr at 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 librados: starting objecter 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 Processor -- start 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 -- start start 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 librados: setting wanted keys 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 librados: calling monclient init 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient: init 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8bb00703b0 0x7f8bb01e97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f8bb00703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f8bb0150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb015c710 0x7f8bb0073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb015c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:35.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0150600 con 0x7f8bb0150220 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb015bae0 con 0x7f8bb00703b0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0150780 con 0x7f8bb015c710 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.589+0000 7f8bb6ec9640 10 monclient(hunting): authenticate will time out at 1251.169929s 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb015c710 0x7f8bb0073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8bb00703b0 0x7f8bb01e97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:59000/0 (socket says 10.20.193.51:59000) 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 1 -- 10.20.193.51:0/1180918663 learned_addr learned my addr 10.20.193.51:0/1180918663 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): get_auth_request con 0x7f8bb0150220 auth_method 0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): get_auth_request con 0x7f8bb015c710 auth_method 0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): get_auth_request con 0x7f8bb00703b0 auth_method 0 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb543f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:35.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.590+0000 7f8bb4c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_done global_id 4839 payload 824 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 -- 10.20.193.51:0/1180918663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb015c710 msgr2=0x7f8bb0073450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb015c710 0x7f8bb0073450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 -- 10.20.193.51:0/1180918663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8bb00703b0 msgr2=0x7f8bb01e97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8bb00703b0 0x7f8bb01e97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 monclient: found mon.noname-a 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.592+0000 7f8ba7fff640 1 -- 10.20.193.51:0/1180918663 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb00c8300 con 0x7f8bb0150220 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.593+0000 7f8ba7fff640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800c730 tx=0x7f8b9800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.593+0000 7f8ba57fa640 10 client.?.objecter ms_handle_connect 0x7f8bb0150220 2026-02-22T21:39:35.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.593+0000 7f8ba57fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98017070 con 0x7f8bb0150220 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: dump: 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:35.596 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: _finish_auth 0 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: _check_auth_tickets 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 5 monclient: authenticate success, global_id 4839 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b98005bb0 con 0x7f8bb0150220 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 10 monclient: _renew_subs 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb00c85e0 con 0x7f8bb0150220 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98007070 con 0x7f8bb0150220 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba6ffd640 4 set_mon_vals no callback set 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8ba57fa640 10 monclient: dump: 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 10 client.4839.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 10 monclient: _renew_subs 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb00c8ba0 con 0x7f8bb0150220 2026-02-22T21:39:35.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.594+0000 7f8bb6ec9640 1 librados: init done 2026-02-22T21:39:35.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.595+0000 7f8bb6ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:35.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.595+0000 7f8bb6ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:35.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.595+0000 7f8bb6ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:35.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.595+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb00c73a0 con 0x7f8bb0150220 2026-02-22T21:39:35.607 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.606+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f8b98007210 con 0x7f8bb0150220 2026-02-22T21:39:35.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.607+0000 7f8ba57fa640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 0x7f8b8007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:35.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.607+0000 7f8bb4c3e640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 0x7f8b8007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:35.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.607+0000 7f8bb4c3e640 10 monclient: get_auth_request con 0x7f8b80078e50 auth_method 0 2026-02-22T21:39:35.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.607+0000 7f8bb4c3e640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 0x7f8b8007b310 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00051f0 tx=0x7f8ba0007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:35.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.607+0000 7f8ba57fa640 10 client.4839.objecter ms_handle_connect 0x7f8b80078e50 2026-02-22T21:39:35.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.608+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f8b9805d4a0 con 0x7f8bb0150220 2026-02-22T21:39:35.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.608+0000 7f8ba57fa640 10 client.4839.objecter ms_dispatch2 0x7f8bb0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:35.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.608+0000 7f8ba57fa640 3 client.4839.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:35.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.608+0000 7f8ba57fa640 3 client.4839.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:35.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.608+0000 7f8ba57fa640 20 client.4839.objecter dump_active .. 0 homeless 2026-02-22T21:39:35.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.709+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f8b98021020 con 0x7f8bb0150220 2026-02-22T21:39:35.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.709+0000 7f8ba57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:35.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.709+0000 7f8ba57fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:35.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.811+0000 7f8bb6ec9640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:35.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.811+0000 7f8bb6ec9640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:35.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.811+0000 7f8bb6ec9640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:35.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.811+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8bb00bfcd0 con 0x7f8bb0150220 2026-02-22T21:39:35.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8ba57fa640 1 -- 10.20.193.51:0/1180918663 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f8b98061460 con 0x7f8bb0150220 2026-02-22T21:39:35.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8ba57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:35.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8ba57fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:35.816 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:35.816 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:35.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8bb6ec9640 10 librados: watch_flush enter 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8bb6ec9640 10 librados: watch_flush exit 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8bb6ec9640 20 client.4839.objecter shutdown clearing up homeless session... 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.814+0000 7f8bb6ec9640 10 client.4839.objecter successfully canceled tick 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 msgr2=0x7f8b8007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 0x7f8b8007b310 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00051f0 tx=0x7f8ba0007f30 comp rx=0 tx=0).stop 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 10 monclient: shutdown 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 msgr2=0x7f8bb00cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800c730 tx=0x7f8b9800cc00 comp rx=0 tx=0).stop 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 shutdown_connections 2026-02-22T21:39:35.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8b80078e50 0x7f8b8007b310 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8bb015c710 0x7f8bb0073450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8bb0150220 0x7f8bb00cc460 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 --2- 10.20.193.51:0/1180918663 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8bb00703b0 0x7f8bb01e97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 >> 10.20.193.51:0/1180918663 conn(0x7f8bb00cd7b0 msgr2=0x7f8bb00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 shutdown_connections 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 -- 10.20.193.51:0/1180918663 wait complete. 2026-02-22T21:39:35.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:35.815+0000 7f8bb6ec9640 1 librados: shutdown 2026-02-22T21:39:38.826 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 10 monclient: get_monmap_and_config 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 10 monclient: build_initial_monmap 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 10 monclient: monmap: 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:38.893175+0000 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:38.893175+0000 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:38.894 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 1 Processor -- start 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 1 -- start start 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.892+0000 7fb2a4f3b640 10 monclient: init 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient: _add_conns ranks=[4,1,3,5,2,0] 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb2a0150220 0x7fb2a00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a0150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb2a0150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb2a015c710 0x7fb2a006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a015c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a015a920 con 0x7fb2a0150600 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fb2a015af60 con 0x7fb2a015c710 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fb2a015b240 con 0x7fb2a0150220 2026-02-22T21:39:38.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb2a4f3b640 10 monclient(hunting): authenticate will time out at 1254.473498s 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb29e575640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb2a0150220 0x7fb2a00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.893+0000 7fb29ed76640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb2a015c710 0x7fb2a006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb2a015c710 0x7fb2a006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39294/0 (socket says 10.20.193.51:39294) 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 1 -- 10.20.193.51:0/4205395634 learned_addr learned my addr 10.20.193.51:0/4205395634 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): get_auth_request con 0x7fb2a015c710 auth_method 0 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): get_auth_request con 0x7fb2a0150220 auth_method 0 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:38.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.894+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 10 monclient(hunting): handle_auth_done global_id 4842 payload 274 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 1 -- 10.20.193.51:0/4205395634 >> v2:10.20.193.68:3300/0 conn(0x7fb2a0150220 msgr2=0x7fb2a00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 1 --2- 10.20.193.51:0/4205395634 >> v2:10.20.193.68:3300/0 conn(0x7fb2a0150220 0x7fb2a00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 1 -- 10.20.193.51:0/4205395634 >> v1:10.20.193.68:6789/0 conn(0x7fb2a0150600 legacy=0x7fb2a015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 1 monclient: found mon.noname-a 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.895+0000 7fb29ed76640 1 -- 10.20.193.51:0/4205395634 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a015b520 con 0x7fb2a015c710 2026-02-22T21:39:38.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.896+0000 7fb29ed76640 1 --2- 10.20.193.51:0/4205395634 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a015c710 0x7fb2a006fca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0070940 tx=0x7fb29002f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6836f2f0be34fe29 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 1 -- 10.20.193.51:0/4205395634 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29002fe10 con 0x7fb2a015c710 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: dump: 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: _finish_auth 0 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: _check_auth_tickets 2026-02-22T21:39:38.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 5 monclient: authenticate success, global_id 4842 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 1 -- 10.20.193.51:0/4205395634 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2900373f0 con 0x7fb2a015c710 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 1 -- 10.20.193.51:0/4205395634 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2900377c0 con 0x7fb2a015c710 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb29d573640 10 monclient: dump: 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 4 set_mon_vals no callback set 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 10 monclient: shutdown 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/4205395634 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a015c710 msgr2=0x7fb2a006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/4205395634 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a015c710 0x7fb2a006fca0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0070940 tx=0x7fb29002f310 comp rx=0 tx=0).stop 2026-02-22T21:39:38.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/4205395634 shutdown_connections 2026-02-22T21:39:38.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/4205395634 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a015c710 0x7fb2a006fca0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:38.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/4205395634 >> v2:10.20.193.68:3300/0 conn(0x7fb2a0150220 0x7fb2a00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:38.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/4205395634 >> 10.20.193.51:0/4205395634 conn(0x7fb2a00cd7b0 msgr2=0x7fb2a00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:38.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/4205395634 shutdown_connections 2026-02-22T21:39:38.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.898+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/4205395634 wait complete. 2026-02-22T21:39:38.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient: build_initial_monmap 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient: monmap: 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 librados: starting msgr at 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 librados: starting objecter 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 Processor -- start 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 -- start start 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 librados: setting wanted keys 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 librados: calling monclient init 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient: init 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 0x7fb2a014c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a0150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2a0150600 0x7fb2a00bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a0150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:38.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 0x7fb2a00bf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a00701e0 con 0x7fb2a0150220 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a015bae0 con 0x7fb2a015c710 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a0070360 con 0x7fb2a0150600 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.899+0000 7fb2a4f3b640 10 monclient(hunting): authenticate will time out at 1254.479953s 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 0x7fb2a014c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 0x7fb2a00bf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 0x7fb2a014c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39298/0 (socket says 10.20.193.51:39298) 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 1 -- 10.20.193.51:0/1973305910 learned_addr learned my addr 10.20.193.51:0/1973305910 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): get_auth_request con 0x7fb2a0150220 auth_method 0 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): get_auth_request con 0x7fb2a015c710 auth_method 0 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29dd74640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2a0150600 0x7fb2a00bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29e575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:38.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29dd74640 10 monclient(hunting): get_auth_request con 0x7fb2a0150600 auth_method 0 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29dd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29dd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.900+0000 7fb29dd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29dd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29dd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29dd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29ed76640 10 monclient(hunting): handle_auth_done global_id 4810 payload 824 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29ed76640 1 -- 10.20.193.51:0/1973305910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2a0150600 msgr2=0x7fb2a00bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29ed76640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2a0150600 0x7fb2a00bf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29ed76640 1 -- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 msgr2=0x7fb2a014c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.901+0000 7fb29ed76640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 0x7fb2a014c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 1 monclient: found mon.noname-b 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:38.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 1 -- 10.20.193.51:0/1973305910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a00bfe00 con 0x7fb2a015c710 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb29ed76640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 0x7fb2a00bf730 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb290035000 tx=0x7fb290004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb286ffd640 10 client.?.objecter ms_handle_connect 0x7fb2a015c710 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.902+0000 7fb286ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290046020 con 0x7fb2a015c710 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: dump: 2026-02-22T21:39:38.907 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: _finish_auth 0 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb290037e30 con 0x7fb2a015c710 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 5 monclient: authenticate success, global_id 4810 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 10 monclient: _renew_subs 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a00c00e0 con 0x7fb2a015c710 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290009510 con 0x7fb2a015c710 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb286ffd640 10 monclient: dump: 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.908 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb29cd72640 4 set_mon_vals no callback set 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 10 client.4810.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 10 monclient: _renew_subs 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a0072a40 con 0x7fb2a015c710 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.904+0000 7fb2a4f3b640 1 librados: init done 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.905+0000 7fb2a4f3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.905+0000 7fb2a4f3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.905+0000 7fb2a4f3b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:38.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:38.905+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a00c46c0 con 0x7fb2a015c710 2026-02-22T21:39:39.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.011+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb290007bf0 con 0x7fb2a015c710 2026-02-22T21:39:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.011+0000 7fb286ffd640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 0x7fb26c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:39.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.011+0000 7fb29e575640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 0x7fb26c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.011+0000 7fb29e575640 10 monclient: get_auth_request con 0x7fb26c078e50 auth_method 0 2026-02-22T21:39:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.012+0000 7fb29e575640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 0x7fb26c07b310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb294009d00 tx=0x7fb294009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:39.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.012+0000 7fb286ffd640 10 client.4810.objecter ms_handle_connect 0x7fb26c078e50 2026-02-22T21:39:39.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.014+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb290083c20 con 0x7fb2a015c710 2026-02-22T21:39:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.014+0000 7fb286ffd640 10 client.4810.objecter ms_dispatch2 0x7fb2a0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.014+0000 7fb286ffd640 3 client.4810.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.014+0000 7fb286ffd640 3 client.4810.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:39.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.014+0000 7fb286ffd640 20 client.4810.objecter dump_active .. 0 homeless 2026-02-22T21:39:39.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.023+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb2a00c46c0 con 0x7fb2a015c710 2026-02-22T21:39:39.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.023+0000 7fb286ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:39.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.023+0000 7fb286ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:39.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.123+0000 7fb2a4f3b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:39.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.123+0000 7fb2a4f3b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:39.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.123+0000 7fb2a4f3b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:39.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.123+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2a006fca0 con 0x7fb2a015c710 2026-02-22T21:39:39.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.125+0000 7fb286ffd640 1 -- 10.20.193.51:0/1973305910 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb290050030 con 0x7fb2a015c710 2026-02-22T21:39:39.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.125+0000 7fb286ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:39.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.125+0000 7fb286ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:39.127 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:39.127 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 10 librados: watch_flush enter 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 10 librados: watch_flush exit 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 20 client.4810.objecter shutdown clearing up homeless session... 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 10 client.4810.objecter successfully canceled tick 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 msgr2=0x7fb26c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 0x7fb26c07b310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb294009d00 tx=0x7fb294009290 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 10 monclient: shutdown 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 msgr2=0x7fb2a00bf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 0x7fb2a00bf730 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb290035000 tx=0x7fb290004b40 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 shutdown_connections 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb26c078e50 0x7fb26c07b310 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2a015c710 0x7fb2a00bf730 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2a0150600 0x7fb2a00bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 --2- 10.20.193.51:0/1973305910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2a0150220 0x7fb2a014c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 >> 10.20.193.51:0/1973305910 conn(0x7fb2a00cd7b0 msgr2=0x7fb2a00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 shutdown_connections 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 -- 10.20.193.51:0/1973305910 wait complete. 2026-02-22T21:39:39.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:39.126+0000 7fb2a4f3b640 1 librados: shutdown 2026-02-22T21:39:42.137 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.200+0000 7f9ff43d3640 10 monclient: get_monmap_and_config 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.200+0000 7f9ff43d3640 10 monclient: build_initial_monmap 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.200+0000 7f9ff43d3640 10 monclient: monmap: 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:42.201463+0000 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:42.201463+0000 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:42.202 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 Processor -- start 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 -- start start 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient: init 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient: _add_conns ranks=[5,4,3,0,2,1] 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9fec150220 0x7f9fec0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9fec150600 0x7f9fec15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 0x7f9fec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec15c540 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec15c9b0 con 0x7f9fec15c540 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec15a920 con 0x7f9fec150600 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec15af60 con 0x7f9fec150220 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:42.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.201+0000 7f9ff43d3640 10 monclient(hunting): authenticate will time out at 1257.781749s 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9fec150220 0x7f9fec0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 0x7f9fec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9fec150600 0x7f9fec15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9fec150220 0x7f9fec0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44354/0 (socket says 10.20.193.51:44354) 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 1 -- 10.20.193.51:0/1823077096 learned_addr learned my addr 10.20.193.51:0/1823077096 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): get_auth_request con 0x7f9fec150220 auth_method 0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): get_auth_request con 0x7f9fec15c540 auth_method 0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): get_auth_request con 0x7f9fec150600 auth_method 0 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.202+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 10 monclient(hunting): handle_auth_done global_id 4781 payload 274 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 -- 10.20.193.51:0/1823077096 >> v2:10.20.193.68:3300/0 conn(0x7f9fec150600 msgr2=0x7f9fec15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 --2- 10.20.193.51:0/1823077096 >> v2:10.20.193.68:3300/0 conn(0x7f9fec150600 0x7f9fec15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 -- 10.20.193.51:0/1823077096 >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 msgr2=0x7f9fec06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 --2- 10.20.193.51:0/1823077096 >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 0x7f9fec06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 10 monclient: _finish_hunting 0 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 monclient: found mon.noname-c 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2148640 1 -- 10.20.193.51:0/1823077096 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec15b240 con 0x7f9fec150220 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.203+0000 7f9ff2949640 1 --2- 10.20.193.51:0/1823077096 >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 0x7f9fec06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:39:42.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.204+0000 7f9ff2148640 1 --2- 10.20.193.51:0/1823077096 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150220 0x7f9fec0c73a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9fec074940 tx=0x7f9fdc02ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ce5ae2142a9d77 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 1 -- 10.20.193.51:0/1823077096 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc03c070 con 0x7f9fec150220 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: dump: 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: _finish_auth 0 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: _check_auth_tickets 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff43d3640 5 monclient: authenticate success, global_id 4781 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff43d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 1 -- 10.20.193.51:0/1823077096 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fdc02fbd0 con 0x7f9fec150220 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 1 -- 10.20.193.51:0/1823077096 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc037440 con 0x7f9fec150220 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff1146640 10 monclient: dump: 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.208 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff43d3640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.206+0000 7f9ff43d3640 4 set_mon_vals no callback set 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 10 monclient: shutdown 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 -- 10.20.193.51:0/1823077096 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150220 msgr2=0x7f9fec0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:42.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/1823077096 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150220 0x7f9fec0c73a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9fec074940 tx=0x7f9fdc02ef50 comp rx=0 tx=0).stop 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 -- 10.20.193.51:0/1823077096 shutdown_connections 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/1823077096 >> v2:10.20.193.51:3300/0 conn(0x7f9fec15c540 0x7f9fec06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/1823077096 >> v2:10.20.193.68:3300/0 conn(0x7f9fec150600 0x7f9fec15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/1823077096 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150220 0x7f9fec0c73a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 -- 10.20.193.51:0/1823077096 >> 10.20.193.51:0/1823077096 conn(0x7f9fec0cd7b0 msgr2=0x7f9fec0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 -- 10.20.193.51:0/1823077096 shutdown_connections 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 -- 10.20.193.51:0/1823077096 wait complete. 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 10 monclient: build_initial_monmap 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 10 monclient: monmap: 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 librados: starting msgr at 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 librados: starting objecter 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.207+0000 7f9ff43d3640 1 Processor -- start 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 -- start start 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 librados: setting wanted keys 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 librados: calling monclient init 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient: init 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9fec150220 0x7f9fec0bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9fec15c540 0x7f9fec14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec15c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec15bae0 con 0x7f9fec150220 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec15b800 con 0x7f9fec15c540 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:42.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec0701e0 con 0x7f9fec150600 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff43d3640 10 monclient(hunting): authenticate will time out at 1257.788431s 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2148640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9fec150220 0x7f9fec0bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2949640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9fec15c540 0x7f9fec14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:44364/0 (socket says 10.20.193.51:44364) 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 1 -- 10.20.193.51:0/2620420261 learned_addr learned my addr 10.20.193.51:0/2620420261 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 10 monclient(hunting): get_auth_request con 0x7f9fec150600 auth_method 0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff1947640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2148640 10 monclient(hunting): get_auth_request con 0x7f9fec150220 auth_method 0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2148640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.208+0000 7f9ff2148640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): get_auth_request con 0x7f9fec15c540 auth_method 0 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff1947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.209+0000 7f9ff2949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 10 monclient(hunting): handle_auth_done global_id 4784 payload 824 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 -- 10.20.193.51:0/2620420261 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9fec15c540 msgr2=0x7f9fec14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9fec15c540 0x7f9fec14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 -- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9fec150220 msgr2=0x7f9fec0bfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9fec150220 0x7f9fec0bfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 10 monclient: _finish_hunting 0 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 monclient: found mon.noname-c 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.210+0000 7f9ff1947640 1 -- 10.20.193.51:0/2620420261 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec0be8b0 con 0x7f9fec150600 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.211+0000 7f9ff1947640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ebe0 tx=0x7f9fe000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.211+0000 7f9fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9fec150600 2026-02-22T21:39:42.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.211+0000 7f9fd2ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0017070 con 0x7f9fec150600 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: dump: 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.214 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe000cc40 con 0x7f9fec150600 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0013470 con 0x7f9fec150600 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff0945640 4 set_mon_vals no callback set 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9fd2ffd640 10 monclient: dump: 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 5 monclient: authenticate success, global_id 4784 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _renew_subs 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec0beb90 con 0x7f9fec150600 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 client.4784.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _renew_subs 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec0bf150 con 0x7f9fec150600 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 1 librados: init done 2026-02-22T21:39:42.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:42.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:42.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.213+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec070570 con 0x7f9fec150600 2026-02-22T21:39:42.226 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.225+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9fe0013610 con 0x7f9fec150600 2026-02-22T21:39:42.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.225+0000 7f9fd2ffd640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 0x7f9fbc07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:42.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.225+0000 7f9ff2148640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 0x7f9fbc07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:42.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.225+0000 7f9ff2148640 10 monclient: get_auth_request con 0x7f9fbc078e30 auth_method 0 2026-02-22T21:39:42.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.226+0000 7f9ff2148640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 0x7f9fbc07b2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0c65c0 tx=0x7f9fdc039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:42.227 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.226+0000 7f9fd2ffd640 10 client.4784.objecter ms_handle_connect 0x7f9fbc078e30 2026-02-22T21:39:42.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.227+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9fe005e4b0 con 0x7f9fec150600 2026-02-22T21:39:42.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.227+0000 7f9fd2ffd640 10 client.4784.objecter ms_dispatch2 0x7f9fec000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:42.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.227+0000 7f9fd2ffd640 3 client.4784.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:42.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.227+0000 7f9fd2ffd640 3 client.4784.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:42.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.227+0000 7f9fd2ffd640 20 client.4784.objecter dump_active .. 0 homeless 2026-02-22T21:39:42.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.328+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9fe0062470 con 0x7f9fec150600 2026-02-22T21:39:42.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.328+0000 7f9fd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:42.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.328+0000 7f9fd2ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:42.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.431+0000 7f9ff43d3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:42.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.431+0000 7f9ff43d3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:42.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.431+0000 7f9ff43d3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:39:42.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.431+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9fec074da0 con 0x7f9fec150600 2026-02-22T21:39:42.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.433+0000 7f9fd2ffd640 1 -- 10.20.193.51:0/2620420261 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f9fe0066770 con 0x7f9fec150600 2026-02-22T21:39:42.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.433+0000 7f9fd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:42.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.433+0000 7f9fd2ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:42.435 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:42.435 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 10 librados: watch_flush enter 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 10 librados: watch_flush exit 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 20 client.4784.objecter shutdown clearing up homeless session... 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 10 client.4784.objecter successfully canceled tick 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 msgr2=0x7f9fbc07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 0x7f9fbc07b2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0c65c0 tx=0x7f9fdc039040 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 10 monclient: shutdown 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 msgr2=0x7f9fec1f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ebe0 tx=0x7f9fe000c520 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 shutdown_connections 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9fbc078e30 0x7f9fbc07b2f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9fec15c540 0x7f9fec14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9fec150600 0x7f9fec1f6cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 --2- 10.20.193.51:0/2620420261 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9fec150220 0x7f9fec0bfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 >> 10.20.193.51:0/2620420261 conn(0x7f9fec0cd7b0 msgr2=0x7f9fec06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.434+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 shutdown_connections 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.435+0000 7f9ff43d3640 1 -- 10.20.193.51:0/2620420261 wait complete. 2026-02-22T21:39:42.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:42.435+0000 7f9ff43d3640 1 librados: shutdown 2026-02-22T21:39:45.444 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.512+0000 7f88d4c16640 10 monclient: get_monmap_and_config 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.512+0000 7f88d4c16640 10 monclient: build_initial_monmap 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.512+0000 7f88d4c16640 10 monclient: monmap: 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:45.514353+0000 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:45.514353+0000 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:45.515 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:45.516 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:45.516 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:45.516 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:45.516 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:45.516 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 1 Processor -- start 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 1 -- start start 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient: init 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient: _add_conns ranks=[2,0,1,5,3,4] 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88d0150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88d015c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f88d00703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d015a920 con 0x7f88d015c710 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d015af60 con 0x7f88d00703b0 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d015b240 con 0x7f88d0150220 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.513+0000 7f88d4c16640 10 monclient(hunting): authenticate will time out at 1261.094877s 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.514+0000 7f88cdd74640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f88d015c710 0x7f88d006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:53536/0 (socket says 10.20.193.51:53536) 2026-02-22T21:39:45.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.514+0000 7f88cdd74640 1 -- 10.20.193.51:0/1949609946 learned_addr learned my addr 10.20.193.51:0/1949609946 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319653612 0 0) 0x7f88d015a920 con 0x7f88d015c710 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _negotiate 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): my global_id is 4854 2026-02-22T21:39:45.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88bc003620 con 0x7f88d015c710 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263373966 0 0) 0x7f88d015af60 con 0x7f88d00703b0 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _negotiate 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 10 monclient(hunting): my global_id is 4819 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.515+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d015a920 con 0x7f88d00703b0 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488277984 0 0) 0x7f88d015b240 con 0x7f88d0150220 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 10 monclient(hunting): _negotiate 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 10 monclient(hunting): my global_id is 4787 2026-02-22T21:39:45.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.516+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d015af60 con 0x7f88d0150220 2026-02-22T21:39:45.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3433025399 0 0) 0x7f88bc003620 con 0x7f88d015c710 2026-02-22T21:39:45.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 >> v1:10.20.193.125:6789/0 conn(0x7f88d0150220 legacy=0x7f88d015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 >> v1:10.20.193.68:6789/0 conn(0x7f88d00703b0 legacy=0x7f88d0159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _finish_hunting 0 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d015b520 con 0x7f88d015c710 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _finish_auth 0 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _check_auth_tickets 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f88b8002f00 con 0x7f88d015c710 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: dump: 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 >> v1:10.20.193.51:6789/0 conn(0x7f88d015c710 legacy=0x7f88d006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88bc003fe0 0x7f88bc004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): picked mon.a con 0x7f88bc003fe0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88bc0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): picked mon.b con 0x7f88bc004e40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88bc009ab0 0x7f88bc00bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): picked mon.c con 0x7f88bc009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc00c490 con 0x7f88bc003fe0 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc00c640 con 0x7f88bc004e40 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc00c840 con 0x7f88bc009ab0 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.517+0000 7f88cd573640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88bc0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): get_auth_request con 0x7f88bc004e40 auth_method 0 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:45.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ced76640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88bc009ab0 0x7f88bc00bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ce575640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88bc003fe0 0x7f88bc004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ced76640 10 monclient(hunting): get_auth_request con 0x7f88bc009ab0 auth_method 0 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ced76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ced76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ced76640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ce575640 10 monclient(hunting): get_auth_request con 0x7f88bc003fe0 auth_method 0 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ce575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ce575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.518+0000 7f88ce575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 10 monclient(hunting): handle_auth_done global_id 4854 payload 293 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 -- 10.20.193.51:0/1949609946 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88bc009ab0 msgr2=0x7f88bc00bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88bc009ab0 0x7f88bc00bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 -- 10.20.193.51:0/1949609946 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88bc003fe0 msgr2=0x7f88bc004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88bc003fe0 0x7f88bc004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 10 monclient: _finish_hunting 0 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 monclient: found mon.b 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88ced76640 1 -- 10.20.193.51:0/1949609946 reap_dead start 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 1 -- 10.20.193.51:0/1949609946 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc003620 con 0x7f88bc004e40 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 10 monclient: _finish_auth 0 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 10 monclient: _check_auth_tickets 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88cdd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88d4c16640 5 monclient: authenticate success, global_id 4854 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.519+0000 7f88d4c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.520+0000 7f88cdd74640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88bc0052c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c240 tx=0x7f88b802f530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5a8eccc7418641 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.521+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007d20 con 0x7f88bc004e40 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.521+0000 7f88cd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.521+0000 7f88cd573640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.521+0000 7f88cd573640 10 monclient: dump: 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.525 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.521+0000 7f88d4c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b8005bf0 con 0x7f88bc004e40 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 1 -- 10.20.193.51:0/1949609946 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007070 con 0x7f88bc004e40 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88cd573640 10 monclient: dump: 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:45.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 4 set_mon_vals no callback set 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 10 monclient: shutdown 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 -- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 msgr2=0x7f88bc0052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88bc0052c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c240 tx=0x7f88b802f530 comp rx=0 tx=0).stop 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 -- 10.20.193.51:0/1949609946 shutdown_connections 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 --2- 10.20.193.51:0/1949609946 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88bc0052c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 -- 10.20.193.51:0/1949609946 >> 10.20.193.51:0/1949609946 conn(0x7f88d00cd7b0 msgr2=0x7f88d00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 -- 10.20.193.51:0/1949609946 shutdown_connections 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.522+0000 7f88d4c16640 1 -- 10.20.193.51:0/1949609946 wait complete. 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient: build_initial_monmap 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient: monmap: 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:45.529 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 librados: starting msgr at 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 librados: starting objecter 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 Processor -- start 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 -- start start 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 librados: setting wanted keys 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 librados: calling monclient init 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient: init 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88d01f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-b con 0x7f88bc004e40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88d00cc470 0x7f88d0159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-c con 0x7f88d00cc470 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 0x7f88d0159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): picked mon.noname-a con 0x7f88d00cc8f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0150600 con 0x7f88d00cc8f0 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f88d015bae0 con 0x7f88bc004e40 2026-02-22T21:39:45.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0150780 con 0x7f88d00cc470 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88d4c16640 10 monclient(hunting): authenticate will time out at 1261.104729s 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.523+0000 7f88ced76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 0x7f88d0159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88d01f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88d00cc470 0x7f88d0159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 0x7f88d0159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39332/0 (socket says 10.20.193.51:39332) 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 1 -- 10.20.193.51:0/4060377668 learned_addr learned my addr 10.20.193.51:0/4060377668 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): get_auth_request con 0x7f88d00cc8f0 auth_method 0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): get_auth_request con 0x7f88bc004e40 auth_method 0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): get_auth_request con 0x7f88d00cc470 auth_method 0 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ce575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88ced76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.524+0000 7f88cdd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 10 monclient(hunting): handle_auth_done global_id 4822 payload 824 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 -- 10.20.193.51:0/4060377668 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88d00cc470 msgr2=0x7f88d0159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88d00cc470 0x7f88d0159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 -- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 msgr2=0x7f88d0159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 0x7f88d0159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 10 monclient: _finish_hunting 0 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 monclient: found mon.noname-b 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.525+0000 7f88ce575640 1 -- 10.20.193.51:0/4060377668 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d01f4a40 con 0x7f88bc004e40 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.526+0000 7f88ce575640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88d01f7af0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88d0075d40 tx=0x7f88c400c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.526+0000 7f88b6ffd640 10 client.?.objecter ms_handle_connect 0x7f88bc004e40 2026-02-22T21:39:45.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.526+0000 7f88b6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c400c9f0 con 0x7f88bc004e40 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: dump: 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:45.533 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c400cb90 con 0x7f88bc004e40 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c400cd80 con 0x7f88bc004e40 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 5 monclient: authenticate success, global_id 4822 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 10 monclient: _renew_subs 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88ccd72640 4 set_mon_vals no callback set 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d01581b0 con 0x7f88bc004e40 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88b6ffd640 10 monclient: dump: 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:45.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 10 client.4822.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 10 monclient: _renew_subs 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d0158770 con 0x7f88bc004e40 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.528+0000 7f88d4c16640 1 librados: init done 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.529+0000 7f88d4c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.529+0000 7f88d4c16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.529+0000 7f88d4c16640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.529+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d0074da0 con 0x7f88bc004e40 2026-02-22T21:39:45.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.547+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f88c4026070 con 0x7f88bc004e40 2026-02-22T21:39:45.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.547+0000 7f88b6ffd640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 0x7f889c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:45.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.547+0000 7f88cdd74640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 0x7f889c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:45.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.547+0000 7f88cdd74640 10 monclient: get_auth_request con 0x7f889c078e80 auth_method 0 2026-02-22T21:39:45.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.548+0000 7f88b6ffd640 10 client.4822.objecter ms_handle_connect 0x7f889c078e80 2026-02-22T21:39:45.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.548+0000 7f88cdd74640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 0x7f889c07b340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8009070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:45.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.644+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f88c405e030 con 0x7f88bc004e40 2026-02-22T21:39:45.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.644+0000 7f88b6ffd640 10 client.4822.objecter ms_dispatch2 0x7f88d0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:45.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.644+0000 7f88b6ffd640 3 client.4822.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:45.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.644+0000 7f88b6ffd640 3 client.4822.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:45.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.645+0000 7f88b6ffd640 20 client.4822.objecter dump_active .. 0 homeless 2026-02-22T21:39:45.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.657+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f88c4023020 con 0x7f88bc004e40 2026-02-22T21:39:45.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.657+0000 7f88b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:45.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.657+0000 7f88b6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:45.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.758+0000 7f88d4c16640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.758+0000 7f88d4c16640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.758+0000 7f88d4c16640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.758+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f88d00c00e0 con 0x7f88bc004e40 2026-02-22T21:39:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.760+0000 7f88b6ffd640 1 -- 10.20.193.51:0/4060377668 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f88c4061ff0 con 0x7f88bc004e40 2026-02-22T21:39:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.760+0000 7f88b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.760+0000 7f88b6ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:45.763 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:45.763 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:45.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 10 librados: watch_flush enter 2026-02-22T21:39:45.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 10 librados: watch_flush exit 2026-02-22T21:39:45.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 20 client.4822.objecter shutdown clearing up homeless session... 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 10 client.4822.objecter successfully canceled tick 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 msgr2=0x7f889c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 0x7f889c07b340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8009070 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 10 monclient: shutdown 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 msgr2=0x7f88d01f7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88d01f7af0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88d0075d40 tx=0x7f88c400c4b0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 shutdown_connections 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f889c078e80 0x7f889c07b340 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88d00cc8f0 0x7f88d0159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88d00cc470 0x7f88d0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 --2- 10.20.193.51:0/4060377668 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88bc004e40 0x7f88d01f7af0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 >> 10.20.193.51:0/4060377668 conn(0x7f88d00cd7b0 msgr2=0x7f88d00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 shutdown_connections 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 -- 10.20.193.51:0/4060377668 wait complete. 2026-02-22T21:39:45.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:45.762+0000 7f88d4c16640 1 librados: shutdown 2026-02-22T21:39:48.774 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.839+0000 7fe368c91640 10 monclient: get_monmap_and_config 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.839+0000 7fe368c91640 10 monclient: build_initial_monmap 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.839+0000 7fe368c91640 10 monclient: monmap: 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:48.841322+0000 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:48.841322+0000 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:48.842 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 Processor -- start 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 -- start start 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient: init 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient: _add_conns ranks=[2,5,0,4,3,1] 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe364150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe36415c710 0x7fe3640c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-c con 0x7fe36415c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe364150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36415a920 con 0x7fe364150600 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36415af60 con 0x7fe364150220 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe36415b240 con 0x7fe36415c710 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.840+0000 7fe368c91640 10 monclient(hunting): authenticate will time out at 1264.421648s 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe362d76640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fe364150600 0x7fe36406fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:60852/0 (socket says 10.20.193.51:60852) 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe362d76640 1 -- 10.20.193.51:0/780682562 learned_addr learned my addr 10.20.193.51:0/780682562 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:48.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe361d74640 1 --2- 10.20.193.51:0/780682562 >> v2:10.20.193.125:3300/0 conn(0x7fe36415c710 0x7fe3640c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe361d74640 10 monclient(hunting): get_auth_request con 0x7fe36415c710 auth_method 0 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe361d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe361d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.841+0000 7fe361d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 1 -- 10.20.193.51:0/780682562 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711811991 0 0) 0x7fe36415a920 con 0x7fe364150600 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 10 monclient(hunting): _negotiate 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 10 monclient(hunting): my global_id is 4860 2026-02-22T21:39:48.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.842+0000 7fe361573640 1 -- 10.20.193.51:0/780682562 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe350003620 con 0x7fe364150600 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 10 monclient(hunting): handle_auth_done global_id 4793 payload 274 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 1 -- 10.20.193.51:0/780682562 >> v1:10.20.193.125:6789/0 conn(0x7fe364150220 legacy=0x7fe36415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 1 -- 10.20.193.51:0/780682562 >> v1:10.20.193.51:6789/0 conn(0x7fe364150600 legacy=0x7fe36406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 10 monclient: _finish_hunting 0 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 1 monclient: found mon.noname-c 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:39:48.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.843+0000 7fe361d74640 1 -- 10.20.193.51:0/780682562 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36415b520 con 0x7fe36415c710 2026-02-22T21:39:48.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.846+0000 7fe361d74640 1 --2- 10.20.193.51:0/780682562 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe36415c710 0x7fe3640c73a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0098d0 tx=0x7fe34c02ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a13c1c025f74ff server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 1 -- 10.20.193.51:0/780682562 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c03c070 con 0x7fe36415c710 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: dump: 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.850 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: _finish_auth 0 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: _check_auth_tickets 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 1 -- 10.20.193.51:0/780682562 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe34c02fbf0 con 0x7fe36415c710 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 5 monclient: authenticate success, global_id 4793 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 1 -- 10.20.193.51:0/780682562 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c037440 con 0x7fe36415c710 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 4 set_mon_vals no callback set 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe361573640 10 monclient: dump: 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:48.851 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 10 monclient: shutdown 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 -- 10.20.193.51:0/780682562 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe36415c710 msgr2=0x7fe3640c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 --2- 10.20.193.51:0/780682562 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe36415c710 0x7fe3640c73a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0098d0 tx=0x7fe34c02ef20 comp rx=0 tx=0).stop 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 -- 10.20.193.51:0/780682562 shutdown_connections 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 --2- 10.20.193.51:0/780682562 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe36415c710 0x7fe3640c73a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 -- 10.20.193.51:0/780682562 >> 10.20.193.51:0/780682562 conn(0x7fe3640cd7b0 msgr2=0x7fe3640cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 -- 10.20.193.51:0/780682562 shutdown_connections 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.848+0000 7fe368c91640 1 -- 10.20.193.51:0/780682562 wait complete. 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 10 monclient: build_initial_monmap 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 10 monclient: monmap: 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 librados: starting msgr at 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 librados: starting objecter 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 Processor -- start 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 -- start start 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 librados: setting wanted keys 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 1 librados: calling monclient init 2026-02-22T21:39:48.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.849+0000 7fe368c91640 10 monclient: init 2026-02-22T21:39:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:39:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-a con 0x7fe364150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe364150600 0x7fe3641588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-c con 0x7fe364150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe36415c710 0x7fe36414ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): picked mon.noname-b con 0x7fe36415c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe3640703b0 con 0x7fe364150220 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe36415bae0 con 0x7fe36415c710 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe364070530 con 0x7fe364150600 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe368c91640 10 monclient(hunting): authenticate will time out at 1264.431303s 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe361d74640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe364150600 0x7fe3641588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:41266/0 (socket says 10.20.193.51:41266) 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 1 -- 10.20.193.51:0/554957175 learned_addr learned my addr 10.20.193.51:0/554957175 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 10 monclient(hunting): get_auth_request con 0x7fe364150220 auth_method 0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe362d76640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe36415c710 0x7fe36414ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe361d74640 10 monclient(hunting): get_auth_request con 0x7fe364150600 auth_method 0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe361d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe361d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.850+0000 7fe361d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): get_auth_request con 0x7fe36415c710 auth_method 0 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe361d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.851+0000 7fe362d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 10 monclient(hunting): handle_auth_done global_id 4863 payload 824 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 -- 10.20.193.51:0/554957175 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe364150600 msgr2=0x7fe3641588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe364150600 0x7fe3641588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 -- 10.20.193.51:0/554957175 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe36415c710 msgr2=0x7fe36414ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe36415c710 0x7fe36414ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 10 monclient: _finish_hunting 0 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 monclient: found mon.noname-a 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.852+0000 7fe362575640 1 -- 10.20.193.51:0/554957175 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3640c7950 con 0x7fe364150220 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.853+0000 7fe362575640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c700 tx=0x7fe35800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:48.855 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.853+0000 7fe34affd640 10 client.?.objecter ms_handle_connect 0x7fe364150220 2026-02-22T21:39:48.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.853+0000 7fe34affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580052d0 con 0x7fe364150220 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: dump: 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:48.857 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: _finish_auth 0 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: _check_auth_tickets 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe358005470 con 0x7fe364150220 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35800a860 con 0x7fe364150220 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 5 monclient: authenticate success, global_id 4863 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _renew_subs 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3640c7c30 con 0x7fe364150220 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe360d72640 4 set_mon_vals no callback set 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe34affd640 10 monclient: dump: 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 client.4863.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _renew_subs 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3640c81f0 con 0x7fe364150220 2026-02-22T21:39:48.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 1 librados: init done 2026-02-22T21:39:48.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:48.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:48.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:48.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.855+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3640c73a0 con 0x7fe364150220 2026-02-22T21:39:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe35800aa00 con 0x7fe364150220 2026-02-22T21:39:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe34affd640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 0x7fe33007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe361d74640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 0x7fe33007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe361d74640 10 monclient: get_auth_request con 0x7fe330078e00 auth_method 0 2026-02-22T21:39:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe361d74640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 0x7fe33007b2c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe34c039000 tx=0x7fe34c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.867+0000 7fe34affd640 10 client.4863.objecter ms_handle_connect 0x7fe330078e00 2026-02-22T21:39:48.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.869+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe35800ad20 con 0x7fe364150220 2026-02-22T21:39:48.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.869+0000 7fe34affd640 10 client.4863.objecter ms_dispatch2 0x7fe364000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:48.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.869+0000 7fe34affd640 3 client.4863.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:48.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.869+0000 7fe34affd640 3 client.4863.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:48.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.869+0000 7fe34affd640 20 client.4863.objecter dump_active .. 0 homeless 2026-02-22T21:39:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.970+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe358062010 con 0x7fe364150220 2026-02-22T21:39:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.970+0000 7fe34affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:48.970+0000 7fe34affd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:49.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.071+0000 7fe368c91640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:49.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.071+0000 7fe368c91640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:49.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.071+0000 7fe368c91640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:49.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.071+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe3640c4270 con 0x7fe364150220 2026-02-22T21:39:49.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.074+0000 7fe34affd640 1 -- 10.20.193.51:0/554957175 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe358066310 con 0x7fe364150220 2026-02-22T21:39:49.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.074+0000 7fe34affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:49.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.074+0000 7fe34affd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:49.076 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:49.076 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 10 librados: watch_flush enter 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 10 librados: watch_flush exit 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 20 client.4863.objecter shutdown clearing up homeless session... 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 10 client.4863.objecter successfully canceled tick 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 msgr2=0x7fe33007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 0x7fe33007b2c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe34c039000 tx=0x7fe34c033000 comp rx=0 tx=0).stop 2026-02-22T21:39:49.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 10 monclient: shutdown 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 msgr2=0x7fe3640c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe35800c700 tx=0x7fe35800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 shutdown_connections 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe330078e00 0x7fe33007b2c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe36415c710 0x7fe36414ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe364150600 0x7fe3641588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 --2- 10.20.193.51:0/554957175 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe364150220 0x7fe3640c89f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 >> 10.20.193.51:0/554957175 conn(0x7fe3640cd7b0 msgr2=0x7fe3640cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 shutdown_connections 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 -- 10.20.193.51:0/554957175 wait complete. 2026-02-22T21:39:49.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:49.075+0000 7fe368c91640 1 librados: shutdown 2026-02-22T21:39:52.086 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:52.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.151+0000 7fcfafbdc640 10 monclient: get_monmap_and_config 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.151+0000 7fcfafbdc640 10 monclient: build_initial_monmap 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.151+0000 7fcfafbdc640 10 monclient: monmap: 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:52.153632+0000 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:52.153632+0000 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:52.157 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 Processor -- start 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 -- start start 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient: init 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient: _add_conns ranks=[1,3,4,2,0,5] 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fcfa8150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcfa815c710 0x7fcfa80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa815c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fcfa8150600 0x7fcfa806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa8150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa815a920 con 0x7fcfa8150220 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa815af60 con 0x7fcfa815c710 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa815b240 con 0x7fcfa8150600 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:52.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.152+0000 7fcfafbdc640 10 monclient(hunting): authenticate will time out at 1267.733920s 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfae152640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fcfa8150600 0x7fcfa806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcfa815c710 0x7fcfa80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcfa815c710 0x7fcfa80c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:41270/0 (socket says 10.20.193.51:41270) 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 1 -- 10.20.193.51:0/167623837 learned_addr learned my addr 10.20.193.51:0/167623837 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): get_auth_request con 0x7fcfa815c710 auth_method 0 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfae152640 10 monclient(hunting): get_auth_request con 0x7fcfa8150600 auth_method 0 2026-02-22T21:39:52.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfae152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfae152640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfae152640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.153+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:52.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 10 monclient(hunting): handle_auth_done global_id 4866 payload 274 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 1 -- 10.20.193.51:0/167623837 >> v2:10.20.193.68:3300/0 conn(0x7fcfa8150600 msgr2=0x7fcfa806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 1 --2- 10.20.193.51:0/167623837 >> v2:10.20.193.68:3300/0 conn(0x7fcfa8150600 0x7fcfa806fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 1 -- 10.20.193.51:0/167623837 >> v1:10.20.193.68:6789/0 conn(0x7fcfa8150220 legacy=0x7fcfa815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 10 monclient: _finish_hunting 0 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 1 monclient: found mon.noname-a 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.154+0000 7fcfad150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.155+0000 7fcfad150640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.155+0000 7fcfad150640 1 -- 10.20.193.51:0/167623837 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa815b520 con 0x7fcfa815c710 2026-02-22T21:39:52.169 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.156+0000 7fcfad150640 1 --2- 10.20.193.51:0/167623837 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa815c710 0x7fcfa80c73a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009970 tx=0x7fcf9c02efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8cfd26264fb5d2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 1 -- 10.20.193.51:0/167623837 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c03c070 con 0x7fcfa815c710 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: dump: 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:52.170 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: _finish_auth 0 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: _check_auth_tickets 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 1 -- 10.20.193.51:0/167623837 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c02fc80 con 0x7fcfa815c710 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 1 -- 10.20.193.51:0/167623837 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c037660 con 0x7fcfa815c710 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfafbdc640 5 monclient: authenticate success, global_id 4866 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfafbdc640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfafbdc640 4 set_mon_vals no callback set 2026-02-22T21:39:52.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.157+0000 7fcfac94f640 10 monclient: dump: 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 10 monclient: shutdown 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- 10.20.193.51:0/167623837 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa815c710 msgr2=0x7fcfa80c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/167623837 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa815c710 0x7fcfa80c73a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009970 tx=0x7fcf9c02efb0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- 10.20.193.51:0/167623837 shutdown_connections 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/167623837 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa815c710 0x7fcfa80c73a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/167623837 >> v2:10.20.193.68:3300/0 conn(0x7fcfa8150600 0x7fcfa806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- 10.20.193.51:0/167623837 >> 10.20.193.51:0/167623837 conn(0x7fcfa80cd7b0 msgr2=0x7fcfa80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- 10.20.193.51:0/167623837 shutdown_connections 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- 10.20.193.51:0/167623837 wait complete. 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 10 monclient: build_initial_monmap 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 10 monclient: monmap: 2026-02-22T21:39:52.173 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 librados: starting msgr at 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 librados: starting objecter 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 Processor -- start 2026-02-22T21:39:52.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 -- start start 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 librados: setting wanted keys 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 1 librados: calling monclient init 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.158+0000 7fcfafbdc640 10 monclient: init 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfa8150220 0x7fcfa814cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa8150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa8150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfa815c710 0x7fcfa806e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa815c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:52.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80701e0 con 0x7fcfa8150600 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa815bae0 con 0x7fcfa815c710 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa8070360 con 0x7fcfa8150220 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfafbdc640 10 monclient(hunting): authenticate will time out at 1267.740377s 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfae152640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfa815c710 0x7fcfa806e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:41286/0 (socket says 10.20.193.51:41286) 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 1 -- 10.20.193.51:0/3453899768 learned_addr learned my addr 10.20.193.51:0/3453899768 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:52.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 10 monclient(hunting): get_auth_request con 0x7fcfa8150600 auth_method 0 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad150640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfad951640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfa8150220 0x7fcfa814cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfae152640 10 monclient(hunting): get_auth_request con 0x7fcfa815c710 auth_method 0 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfae152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfae152640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.159+0000 7fcfae152640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:52.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): get_auth_request con 0x7fcfa8150220 auth_method 0 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfae152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.160+0000 7fcfad951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 10 monclient(hunting): handle_auth_done global_id 4869 payload 824 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 -- 10.20.193.51:0/3453899768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfa8150220 msgr2=0x7fcfa814cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfa8150220 0x7fcfa814cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 -- 10.20.193.51:0/3453899768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfa815c710 msgr2=0x7fcfa806e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfa815c710 0x7fcfa806e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 10 monclient: _finish_hunting 0 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 monclient: found mon.noname-a 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.161+0000 7fcfad150640 1 -- 10.20.193.51:0/3453899768 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa80c42b0 con 0x7fcfa8150600 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.162+0000 7fcfad150640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c02f560 tx=0x7fcf9c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.162+0000 7fcf967fc640 10 client.?.objecter ms_handle_connect 0x7fcfa8150600 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.162+0000 7fcf967fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c047070 con 0x7fcfa8150600 2026-02-22T21:39:52.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: dump: 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:52.180 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: _finish_auth 0 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: _check_auth_tickets 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c004030 con 0x7fcfa8150600 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 5 monclient: authenticate success, global_id 4869 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 10 monclient: _renew_subs 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa80c2f30 con 0x7fcfa8150600 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c03c040 con 0x7fcfa8150600 2026-02-22T21:39:52.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf967fc640 10 monclient: dump: 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:52.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcf97fff640 4 set_mon_vals no callback set 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 10 client.4869.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 10 monclient: _renew_subs 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa80c34f0 con 0x7fcfa8150600 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.163+0000 7fcfafbdc640 1 librados: init done 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.164+0000 7fcfafbdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.164+0000 7fcfafbdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.164+0000 7fcfafbdc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.164+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa806fca0 con 0x7fcfa8150600 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.175+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcf9c004420 con 0x7fcfa8150600 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.176+0000 7fcf967fc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 0x7fcf7807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.176+0000 7fcfad951640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 0x7fcf7807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.176+0000 7fcfad951640 10 monclient: get_auth_request con 0x7fcf78078e50 auth_method 0 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.176+0000 7fcf967fc640 10 client.4869.objecter ms_handle_connect 0x7fcf78078e50 2026-02-22T21:39:52.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.176+0000 7fcfad951640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 0x7fcf7807b310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfa814d9f0 tx=0x7fcf98007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:52.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.178+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fcf9c082b50 con 0x7fcfa8150600 2026-02-22T21:39:52.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.178+0000 7fcf967fc640 10 client.4869.objecter ms_dispatch2 0x7fcfa8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:52.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.178+0000 7fcf967fc640 3 client.4869.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:52.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.178+0000 7fcf967fc640 3 client.4869.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:52.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.178+0000 7fcf967fc640 20 client.4869.objecter dump_active .. 0 homeless 2026-02-22T21:39:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.280+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fcf9c051020 con 0x7fcfa8150600 2026-02-22T21:39:52.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.280+0000 7fcf967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:52.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.280+0000 7fcf967fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:52.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.384+0000 7fcfafbdc640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:52.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.384+0000 7fcfafbdc640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:52.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.384+0000 7fcfafbdc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:39:52.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.384+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcfa8074da0 con 0x7fcfa8150600 2026-02-22T21:39:52.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.386+0000 7fcf967fc640 1 -- 10.20.193.51:0/3453899768 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fcfa80c42b0 con 0x7fcfa8150600 2026-02-22T21:39:52.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.386+0000 7fcf967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:52.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.386+0000 7fcf967fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:52.389 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:52.389 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:52.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 10 librados: watch_flush enter 2026-02-22T21:39:52.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 10 librados: watch_flush exit 2026-02-22T21:39:52.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 20 client.4869.objecter shutdown clearing up homeless session... 2026-02-22T21:39:52.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 10 client.4869.objecter successfully canceled tick 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 msgr2=0x7fcf7807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 0x7fcf7807b310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfa814d9f0 tx=0x7fcf98007f30 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 10 monclient: shutdown 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 msgr2=0x7fcfa81f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c02f560 tx=0x7fcf9c0047c0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 shutdown_connections 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf78078e50 0x7fcf7807b310 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfa815c710 0x7fcfa806e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfa8150600 0x7fcfa81f2e00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 --2- 10.20.193.51:0/3453899768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfa8150220 0x7fcfa814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:52.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 >> 10.20.193.51:0/3453899768 conn(0x7fcfa80cd7b0 msgr2=0x7fcfa80cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:52.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 shutdown_connections 2026-02-22T21:39:52.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 -- 10.20.193.51:0/3453899768 wait complete. 2026-02-22T21:39:52.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:52.388+0000 7fcfafbdc640 1 librados: shutdown 2026-02-22T21:39:55.399 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.466+0000 7fe67b657640 10 monclient: get_monmap_and_config 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.466+0000 7fe67b657640 10 monclient: build_initial_monmap 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.466+0000 7fe67b657640 10 monclient: monmap: 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:55.468077+0000 2026-02-22T21:39:55.469 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:55.468077+0000 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:55.470 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.466+0000 7fe67b657640 1 Processor -- start 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 1 -- start start 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient: init 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient: _add_conns ranks=[2,4,1,0,3,5] 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6741526a0 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe674152a80 0x7fe6740cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-b con 0x7fe674152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe6740cdb20 addr v1:10.20.193.68:6789/0 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe674073ae0 con 0x7fe6740cdb20 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe674073d60 con 0x7fe6741526a0 2026-02-22T21:39:55.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe6740cca00 con 0x7fe674152a80 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.467+0000 7fe67b657640 10 monclient(hunting): authenticate will time out at 1271.048706s 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe6793cc640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fe6741526a0 0x7fe6740733d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:56350/0 (socket says 10.20.193.51:56350) 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe6793cc640 1 -- 10.20.193.51:0/1186252995 learned_addr learned my addr 10.20.193.51:0/1186252995 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 1 --2- 10.20.193.51:0/1186252995 >> v2:10.20.193.68:3300/0 conn(0x7fe674152a80 0x7fe6740cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): get_auth_request con 0x7fe674152a80 auth_method 0 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:55.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.468+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 1 -- 10.20.193.51:0/1186252995 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779152565 0 0) 0x7fe674073d60 con 0x7fe6741526a0 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 10 monclient(hunting): _negotiate 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 10 monclient(hunting): my global_id is 4805 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.469+0000 7fe663fff640 1 -- 10.20.193.51:0/1186252995 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe65c003620 con 0x7fe6741526a0 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 10 monclient(hunting): handle_auth_done global_id 4837 payload 274 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 1 -- 10.20.193.51:0/1186252995 >> v1:10.20.193.125:6789/0 conn(0x7fe6741526a0 legacy=0x7fe6740733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 1 -- 10.20.193.51:0/1186252995 >> v1:10.20.193.68:6789/0 conn(0x7fe6740cdb20 legacy=0x7fe67415d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 10 monclient: _finish_hunting 0 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 1 monclient: found mon.noname-b 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.470+0000 7fe678bcb640 1 -- 10.20.193.51:0/1186252995 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe674074040 con 0x7fe674152a80 2026-02-22T21:39:55.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.472+0000 7fe678bcb640 1 --2- 10.20.193.51:0/1186252995 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe6740cd5e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe6680098a0 tx=0x7fe66802ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71d46e929c36924 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 1 -- 10.20.193.51:0/1186252995 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66803c070 con 0x7fe674152a80 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: dump: 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: _finish_auth 0 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: _check_auth_tickets 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 1 -- 10.20.193.51:0/1186252995 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe66802fc10 con 0x7fe674152a80 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:55.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 1 -- 10.20.193.51:0/1186252995 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668037440 con 0x7fe674152a80 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe663fff640 10 monclient: dump: 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe67b657640 5 monclient: authenticate success, global_id 4837 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe67b657640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:55.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.474+0000 7fe67b657640 4 set_mon_vals no callback set 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 10 monclient: shutdown 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 -- 10.20.193.51:0/1186252995 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 msgr2=0x7fe6740cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 --2- 10.20.193.51:0/1186252995 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe6740cd5e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe6680098a0 tx=0x7fe66802ef00 comp rx=0 tx=0).stop 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 -- 10.20.193.51:0/1186252995 shutdown_connections 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 --2- 10.20.193.51:0/1186252995 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe6740cd5e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 -- 10.20.193.51:0/1186252995 >> 10.20.193.51:0/1186252995 conn(0x7fe6740ce350 msgr2=0x7fe6740ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 -- 10.20.193.51:0/1186252995 shutdown_connections 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 1 -- 10.20.193.51:0/1186252995 wait complete. 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 10 monclient: build_initial_monmap 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.475+0000 7fe67b657640 10 monclient: monmap: 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 librados: starting msgr at 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 librados: starting objecter 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 Processor -- start 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 -- start start 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 librados: setting wanted keys 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 librados: calling monclient init 2026-02-22T21:39:55.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient: init 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe6740cdb20 0x7fe6741f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-c con 0x7fe6740cdb20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 0x7fe67415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-a con 0x7fe6741526a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe674157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): picked mon.noname-b con 0x7fe674152a80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe674074600 con 0x7fe6741526a0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe674074c50 con 0x7fe674152a80 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe6740748e0 con 0x7fe6740cdb20 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.476+0000 7fe67b657640 10 monclient(hunting): authenticate will time out at 1271.057672s 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 0x7fe67415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe674157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe6740cdb20 0x7fe6741f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 0x7fe67415c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:41292/0 (socket says 10.20.193.51:41292) 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 1 -- 10.20.193.51:0/483145539 learned_addr learned my addr 10.20.193.51:0/483145539 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): get_auth_request con 0x7fe6741526a0 auth_method 0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): get_auth_request con 0x7fe674152a80 auth_method 0 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:55.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): get_auth_request con 0x7fe6740cdb20 auth_method 0 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe678bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe679bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.477+0000 7fe6793cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 10 monclient(hunting): handle_auth_done global_id 4843 payload 824 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 -- 10.20.193.51:0/483145539 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe6740cdb20 msgr2=0x7fe6741f6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe6740cdb20 0x7fe6741f6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 -- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 msgr2=0x7fe67415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 0x7fe67415c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 10 monclient: _finish_hunting 0 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 monclient: found mon.noname-b 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.478+0000 7fe679bcd640 1 -- 10.20.193.51:0/483145539 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6741f5600 con 0x7fe674152a80 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.479+0000 7fe679bcd640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe674157cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe67000cae0 tx=0x7fe67000a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.479+0000 7fe661ffb640 10 client.?.objecter ms_handle_connect 0x7fe674152a80 2026-02-22T21:39:55.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.479+0000 7fe661ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67000acb0 con 0x7fe674152a80 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: dump: 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: _finish_auth 0 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe67000ae50 con 0x7fe674152a80 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670003c90 con 0x7fe674152a80 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe6637fe640 4 set_mon_vals no callback set 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe661ffb640 10 monclient: dump: 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:55.484 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 5 monclient: authenticate success, global_id 4843 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 10 monclient: _renew_subs 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6741f58e0 con 0x7fe674152a80 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 10 client.4843.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 10 monclient: _renew_subs 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6741f5ea0 con 0x7fe674152a80 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.481+0000 7fe67b657640 1 librados: init done 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.482+0000 7fe67b657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.482+0000 7fe67b657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.482+0000 7fe67b657640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.482+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe67406a0b0 con 0x7fe674152a80 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.493+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe6700050d0 con 0x7fe674152a80 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.494+0000 7fe661ffb640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 0x7fe64407b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.494+0000 7fe6793cc640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 0x7fe64407b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.494+0000 7fe6793cc640 10 monclient: get_auth_request con 0x7fe644078e30 auth_method 0 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.494+0000 7fe661ffb640 10 client.4843.objecter ms_handle_connect 0x7fe644078e30 2026-02-22T21:39:55.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.494+0000 7fe6793cc640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 0x7fe64407b2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe664007b60 tx=0x7fe664007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:55.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.496+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe67005e5a0 con 0x7fe674152a80 2026-02-22T21:39:55.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.496+0000 7fe661ffb640 10 client.4843.objecter ms_dispatch2 0x7fe674000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:55.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.496+0000 7fe661ffb640 3 client.4843.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:55.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.496+0000 7fe661ffb640 3 client.4843.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:55.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.496+0000 7fe661ffb640 20 client.4843.objecter dump_active .. 0 homeless 2026-02-22T21:39:55.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.600+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe67406a0b0 con 0x7fe674152a80 2026-02-22T21:39:55.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.600+0000 7fe661ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:55.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.600+0000 7fe661ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:55.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.700+0000 7fe67b657640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:55.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.700+0000 7fe67b657640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:55.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.700+0000 7fe67b657640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:55.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.700+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe6740bb6f0 con 0x7fe674152a80 2026-02-22T21:39:55.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe661ffb640 1 -- 10.20.193.51:0/483145539 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe6740bb6f0 con 0x7fe674152a80 2026-02-22T21:39:55.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe661ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:55.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe661ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:55.704 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:55.704 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:55.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 10 librados: watch_flush enter 2026-02-22T21:39:55.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 10 librados: watch_flush exit 2026-02-22T21:39:55.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 20 client.4843.objecter shutdown clearing up homeless session... 2026-02-22T21:39:55.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 10 client.4843.objecter successfully canceled tick 2026-02-22T21:39:55.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 msgr2=0x7fe64407b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 0x7fe64407b2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe664007b60 tx=0x7fe664007ab0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 10 monclient: shutdown 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 msgr2=0x7fe674157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe674157cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe67000cae0 tx=0x7fe67000a060 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 shutdown_connections 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe644078e30 0x7fe64407b2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe674152a80 0x7fe674157cd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe6741526a0 0x7fe67415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 --2- 10.20.193.51:0/483145539 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe6740cdb20 0x7fe6741f6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 >> 10.20.193.51:0/483145539 conn(0x7fe6740ce350 msgr2=0x7fe67406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 shutdown_connections 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 -- 10.20.193.51:0/483145539 wait complete. 2026-02-22T21:39:55.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:55.703+0000 7fe67b657640 1 librados: shutdown 2026-02-22T21:39:58.715 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: get_monmap_and_config 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: build_initial_monmap 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: monmap: 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:39:58.782751+0000 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:39:58.782751+0000 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:58.783 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 1 Processor -- start 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 1 -- start start 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: init 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 10 monclient: _add_conns ranks=[5,4,2,3,0,1] 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.781+0000 7ffb0ee96640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ffb08152ab0 0x7ffb080c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-c con 0x7ffb08152ab0 addr v2:10.20.193.125:3300/0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ffb0806d9d0 0x7ffb080c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-b con 0x7ffb0806d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb080c5ff0 addr v1:10.20.193.125:6789/0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb0806f1d0 con 0x7ffb080c5ff0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0806ed40 con 0x7ffb0806d9d0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0806de50 con 0x7ffb08152ab0 2026-02-22T21:39:58.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:58.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0ee96640 10 monclient(hunting): authenticate will time out at 1274.363043s 2026-02-22T21:39:58.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ffb0806d9d0 0x7ffb080c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0cc0b640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ffb08152ab0 0x7ffb080c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ffb0806d9d0 0x7ffb080c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:37684/0 (socket says 10.20.193.51:37684) 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 1 -- 10.20.193.51:0/30085738 learned_addr learned my addr 10.20.193.51:0/30085738 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 10 monclient(hunting): get_auth_request con 0x7ffb0806d9d0 auth_method 0 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0cc0b640 10 monclient(hunting): get_auth_request con 0x7ffb08152ab0 auth_method 0 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0cc0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0cc0b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.782+0000 7ffb0cc0b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:58.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.783+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 10 monclient(hunting): handle_auth_done global_id 4846 payload 274 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 1 -- 10.20.193.51:0/30085738 >> v2:10.20.193.125:3300/0 conn(0x7ffb08152ab0 msgr2=0x7ffb080c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 1 --2- 10.20.193.51:0/30085738 >> v2:10.20.193.125:3300/0 conn(0x7ffb08152ab0 0x7ffb080c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 1 -- 10.20.193.51:0/30085738 >> v1:10.20.193.125:6789/0 conn(0x7ffb080c5ff0 legacy=0x7ffb0806fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 1 monclient: found mon.noname-b 2026-02-22T21:39:58.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:58.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:58.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.784+0000 7ffafffff640 1 -- 10.20.193.51:0/30085738 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0806e130 con 0x7ffb0806d9d0 2026-02-22T21:39:58.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.785+0000 7ffafffff640 1 --2- 10.20.193.51:0/30085738 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb0806d9d0 0x7ffb080c8e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffb0806eec0 tx=0x7ffaf002ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1541f95ee7df882 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:58.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 1 -- 10.20.193.51:0/30085738 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf003c070 con 0x7ffb0806d9d0 2026-02-22T21:39:58.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:58.795 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: dump: 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.796 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: _finish_auth 0 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 5 monclient: authenticate success, global_id 4846 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 1 -- 10.20.193.51:0/30085738 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf002fc60 con 0x7ffb0806d9d0 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 1 -- 10.20.193.51:0/30085738 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0037660 con 0x7ffb0806d9d0 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 10 monclient: get_monmap_and_config success 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 4 set_mon_vals no callback set 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:58.797 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffaff7fe640 10 monclient: dump: 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 10 monclient: shutdown 2026-02-22T21:39:58.798 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 -- 10.20.193.51:0/30085738 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb0806d9d0 msgr2=0x7ffb080c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/30085738 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb0806d9d0 0x7ffb080c8e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffb0806eec0 tx=0x7ffaf002ef90 comp rx=0 tx=0).stop 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 -- 10.20.193.51:0/30085738 shutdown_connections 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/30085738 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb0806d9d0 0x7ffb080c8e90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/30085738 >> v2:10.20.193.125:3300/0 conn(0x7ffb08152ab0 0x7ffb080c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 -- 10.20.193.51:0/30085738 >> 10.20.193.51:0/30085738 conn(0x7ffb080cd7b0 msgr2=0x7ffb080cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 -- 10.20.193.51:0/30085738 shutdown_connections 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.787+0000 7ffb0ee96640 1 -- 10.20.193.51:0/30085738 wait complete. 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient: build_initial_monmap 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient: monmap: 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:39:58.799 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 librados: starting msgr at 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 librados: starting objecter 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 Processor -- start 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 -- start start 2026-02-22T21:39:58.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 librados: setting wanted keys 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 librados: calling monclient init 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient: init 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient: _reopen_session rank -1 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 0x7ffb08153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-a con 0x7ffb0806d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 0x7ffb081f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-b con 0x7ffb080c5ff0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffb08152ab0 0x7ffb08073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): picked mon.noname-c con 0x7ffb08152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:58.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0806e9d0 con 0x7ffb0806d9d0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0806e6f0 con 0x7ffb080c5ff0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): start opening mon connection 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ffb08152e90 con 0x7ffb08152ab0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): _renew_subs 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.788+0000 7ffb0ee96640 10 monclient(hunting): authenticate will time out at 1274.369838s 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 0x7ffb08153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 0x7ffb081f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffb08152ab0 0x7ffb08073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 0x7ffb08153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:53882/0 (socket says 10.20.193.51:53882) 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 1 -- 10.20.193.51:0/2670942911 learned_addr learned my addr 10.20.193.51:0/2670942911 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:39:58.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): get_auth_request con 0x7ffb0806d9d0 auth_method 0 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): get_auth_request con 0x7ffb080c5ff0 auth_method 0 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): get_auth_request con 0x7ffb08152ab0 auth_method 0 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:58.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0cc0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.789+0000 7ffb0d40c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 10 monclient(hunting): handle_auth_done global_id 4849 payload 824 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 -- 10.20.193.51:0/2670942911 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffb08152ab0 msgr2=0x7ffb08073cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffb08152ab0 0x7ffb08073cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 -- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 msgr2=0x7ffb08153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 0x7ffb08153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:58.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 10 monclient: _finish_hunting 0 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 monclient: found mon.noname-b 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.791+0000 7ffafffff640 1 -- 10.20.193.51:0/2670942911 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb08072650 con 0x7ffb080c5ff0 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.792+0000 7ffafffff640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 0x7ffb081f75d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffb0806f3c0 tx=0x7ffaf00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.792+0000 7ffafd7fa640 10 client.?.objecter ms_handle_connect 0x7ffb080c5ff0 2026-02-22T21:39:58.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.792+0000 7ffafd7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0047070 con 0x7ffb080c5ff0 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: dump: 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.807 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: _finish_auth 0 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf0004030 con 0x7ffb080c5ff0 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 5 monclient: authenticate success, global_id 4849 2026-02-22T21:39:58.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 10 monclient: _renew_subs 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb08072930 con 0x7ffb080c5ff0 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf003c040 con 0x7ffb080c5ff0 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafd7fa640 10 monclient: dump: 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:39:58.809 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffafeffd640 4 set_mon_vals no callback set 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 10 client.4849.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 10 monclient: _renew_subs 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb08072ef0 con 0x7ffb080c5ff0 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.793+0000 7ffb0ee96640 1 librados: init done 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.794+0000 7ffb0ee96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:58.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.794+0000 7ffb0ee96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:58.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.794+0000 7ffb0ee96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:58.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.794+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb080c5ab0 con 0x7ffb080c5ff0 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ffaf0004420 con 0x7ffb080c5ff0 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffafd7fa640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 0x7ffad807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffb0cc0b640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 0x7ffad807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffb0cc0b640 10 monclient: get_auth_request con 0x7ffad8078e30 auth_method 0 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffb0cc0b640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 0x7ffad807b2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009c60 tx=0x7ffaf8009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:39:58.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.806+0000 7ffafd7fa640 10 client.4849.objecter ms_handle_connect 0x7ffad8078e30 2026-02-22T21:39:58.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.905+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ffaf0038150 con 0x7ffb080c5ff0 2026-02-22T21:39:58.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.905+0000 7ffafd7fa640 10 client.4849.objecter ms_dispatch2 0x7ffb08000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:39:58.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.905+0000 7ffafd7fa640 3 client.4849.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:39:58.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.905+0000 7ffafd7fa640 3 client.4849.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:39:58.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.905+0000 7ffafd7fa640 20 client.4849.objecter dump_active .. 0 homeless 2026-02-22T21:39:58.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.912+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ffaf0086950 con 0x7ffb080c5ff0 2026-02-22T21:39:58.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.912+0000 7ffafd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:39:58.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:58.912+0000 7ffafd7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:39:59.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.012+0000 7ffb0ee96640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:39:59.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.012+0000 7ffb0ee96640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:59.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.012+0000 7ffb0ee96640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:39:59.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.012+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ffb080c65b0 con 0x7ffb080c5ff0 2026-02-22T21:39:59.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffafd7fa640 1 -- 10.20.193.51:0/2670942911 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ffaf008ac50 con 0x7ffb080c5ff0 2026-02-22T21:39:59.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffafd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:39:59.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffafd7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:39:59.017 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:39:59.017 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:39:59.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 10 librados: watch_flush enter 2026-02-22T21:39:59.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 10 librados: watch_flush exit 2026-02-22T21:39:59.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 20 client.4849.objecter shutdown clearing up homeless session... 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 10 client.4849.objecter successfully canceled tick 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 msgr2=0x7ffad807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 0x7ffad807b2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009c60 tx=0x7ffaf8009290 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 10 monclient: shutdown 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 msgr2=0x7ffb081f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 0x7ffb081f75d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffb0806f3c0 tx=0x7ffaf00047c0 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 shutdown_connections 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ffad8078e30 0x7ffad807b2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ffb08152ab0 0x7ffb08073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ffb080c5ff0 0x7ffb081f75d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 --2- 10.20.193.51:0/2670942911 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ffb0806d9d0 0x7ffb08153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 >> 10.20.193.51:0/2670942911 conn(0x7ffb080cd7b0 msgr2=0x7ffb0814ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 shutdown_connections 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 -- 10.20.193.51:0/2670942911 wait complete. 2026-02-22T21:39:59.019 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:39:59.015+0000 7ffb0ee96640 1 librados: shutdown 2026-02-22T21:40:01.469 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:32.623 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.471 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:32.625 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.481 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:32.635 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.499 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:32.650 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.499 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:32.615 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.500 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:32.651 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.500 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:32.616 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.509 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:32.661 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:01.510 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:32.626 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:02.026 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 10 monclient: get_monmap_and_config 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 10 monclient: build_initial_monmap 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 10 monclient: monmap: 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:02.097 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:02.095900+0000 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:02.095900+0000 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 1 Processor -- start 2026-02-22T21:40:02.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 1 -- start start 2026-02-22T21:40:02.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.095+0000 7f8dd20a0640 10 monclient: init 2026-02-22T21:40:02.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient: _add_conns ranks=[1,5,0,4,3,2] 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f8dcc152ab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f8dcc06d9d0 0x7f8dcc0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8dcc06d9d0 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dcc0c93d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dcc06f1d0 con 0x7f8dcc0c93d0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dcc06de50 con 0x7f8dcc152ab0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f8dcc06ed40 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dd20a0640 10 monclient(hunting): authenticate will time out at 1277.676281s 2026-02-22T21:40:02.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dcbfff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f8dcc0c93d0 0x7f8dcc158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:55434/0 (socket says 10.20.193.51:55434) 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dcbfff640 1 -- 10.20.193.51:0/3914390934 learned_addr learned my addr 10.20.193.51:0/3914390934 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.096+0000 7f8dcaffd640 1 --2- 10.20.193.51:0/3914390934 >> v2:10.20.193.125:3300/0 conn(0x7f8dcc06d9d0 0x7f8dcc0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): get_auth_request con 0x7f8dcc06d9d0 auth_method 0 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.097+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110156327 0 0) 0x7f8dcc06de50 con 0x7f8dcc152ab0 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:40:02.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): my global_id is 4852 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8db0003620 con 0x7f8dcc152ab0 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259696973 0 0) 0x7f8dcc06f1d0 con 0x7f8dcc0c93d0 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 10 monclient(hunting): my global_id is 4878 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dcc06de50 con 0x7f8dcc0c93d0 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_done global_id 4820 payload 274 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 1 -- 10.20.193.51:0/3914390934 >> v1:10.20.193.68:6789/0 conn(0x7f8dcc152ab0 legacy=0x7f8dcc0c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 1 -- 10.20.193.51:0/3914390934 >> v1:10.20.193.51:6789/0 conn(0x7f8dcc0c93d0 legacy=0x7f8dcc158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 10 monclient: _finish_hunting 0 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 1 monclient: found mon.noname-c 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.098+0000 7f8dcaffd640 1 -- 10.20.193.51:0/3914390934 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc06e130 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.099+0000 7f8dcaffd640 1 --2- 10.20.193.51:0/3914390934 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc0c8e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0098d0 tx=0x7f8dbc02ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c392f92dbe01a8c0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:02.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc03c070 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: dump: 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.113 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dbc02fbf0 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 1 -- 10.20.193.51:0/3914390934 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc037440 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:02.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dca7fc640 10 monclient: dump: 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dd20a0640 5 monclient: authenticate success, global_id 4820 2026-02-22T21:40:02.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dd20a0640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.101+0000 7f8dd20a0640 4 set_mon_vals no callback set 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 10 monclient: shutdown 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3914390934 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 msgr2=0x7f8dcc0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3914390934 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc0c8e90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0098d0 tx=0x7f8dbc02ef20 comp rx=0 tx=0).stop 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3914390934 shutdown_connections 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3914390934 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc0c8e90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3914390934 >> 10.20.193.51:0/3914390934 conn(0x7f8dcc0cd7b0 msgr2=0x7f8dcc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3914390934 shutdown_connections 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3914390934 wait complete. 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 10 monclient: build_initial_monmap 2026-02-22T21:40:02.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 10 monclient: monmap: 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 librados: starting msgr at 2026-02-22T21:40:02.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 librados: starting objecter 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 Processor -- start 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 -- start start 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 librados: setting wanted keys 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 1 librados: calling monclient init 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.102+0000 7f8dd20a0640 10 monclient: init 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc1f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8dcc06d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dcc0c93d0 0x7f8dcc1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-b con 0x7f8dcc0c93d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:02.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8dcc152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc06e6f0 con 0x7f8dcc152ab0 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0c9880 con 0x7f8dcc0c93d0 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc06e9d0 con 0x7f8dcc06d9d0 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dd20a0640 10 monclient(hunting): authenticate will time out at 1277.683332s 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:02.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcb7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc1f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:53886/0 (socket says 10.20.193.51:53886) 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 1 -- 10.20.193.51:0/3994246047 learned_addr learned my addr 10.20.193.51:0/3994246047 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 10 monclient(hunting): get_auth_request con 0x7f8dcc152ab0 auth_method 0 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcaffd640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dcc0c93d0 0x7f8dcc1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dcc06d9d0 auth_method 0 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.103+0000 7f8dcb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): get_auth_request con 0x7f8dcc0c93d0 auth_method 0 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:02.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.104+0000 7f8dcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 10 monclient(hunting): handle_auth_done global_id 4881 payload 824 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 -- 10.20.193.51:0/3994246047 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 msgr2=0x7f8dcc1f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc1f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 -- 10.20.193.51:0/3994246047 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dcc0c93d0 msgr2=0x7f8dcc1f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dcc0c93d0 0x7f8dcc1f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 monclient: found mon.noname-a 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.105+0000 7f8dcbfff640 1 -- 10.20.193.51:0/3994246047 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc1f3c30 con 0x7f8dcc152ab0 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.106+0000 7f8dcbfff640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000cb40 tx=0x7f8dc000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:02.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.106+0000 7f8da7fff640 10 client.?.objecter ms_handle_connect 0x7f8dcc152ab0 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.106+0000 7f8da7fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000ad50 con 0x7f8dcc152ab0 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: dump: 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:02.123 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc0012400 con 0x7f8dcc152ab0 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 5 monclient: authenticate success, global_id 4881 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _renew_subs 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:02.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc1f3f10 con 0x7f8dcc152ab0 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc00127f0 con 0x7f8dcc152ab0 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dc9ffb640 4 set_mon_vals no callback set 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8da7fff640 10 monclient: dump: 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:02.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 client.4881.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _renew_subs 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:02.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc0c2ec0 con 0x7f8dcc152ab0 2026-02-22T21:40:02.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 1 librados: init done 2026-02-22T21:40:02.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:02.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:02.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:02.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.108+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc152e90 con 0x7f8dcc152ab0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.120+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f8dc0012990 con 0x7f8dcc152ab0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.121+0000 7f8da7fff640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 0x7f8d9c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.121+0000 7f8dcb7fe640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 0x7f8d9c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.121+0000 7f8dcb7fe640 10 monclient: get_auth_request con 0x7f8d9c078e50 auth_method 0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.121+0000 7f8dcb7fe640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 0x7f8d9c07b310 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8db8002ab0 tx=0x7f8db8002a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.121+0000 7f8da7fff640 10 client.4881.objecter ms_handle_connect 0x7f8d9c078e50 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.123+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f8dc005e930 con 0x7f8dcc152ab0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.123+0000 7f8da7fff640 10 client.4881.objecter ms_dispatch2 0x7f8dcc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.123+0000 7f8da7fff640 3 client.4881.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:02.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.123+0000 7f8da7fff640 3 client.4881.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:02.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.123+0000 7f8da7fff640 20 client.4881.objecter dump_active .. 0 homeless 2026-02-22T21:40:02.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.227+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f8dc00628f0 con 0x7f8dcc152ab0 2026-02-22T21:40:02.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.227+0000 7f8da7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:02.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.227+0000 7f8da7fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:02.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.328+0000 7f8dd20a0640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:02.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.328+0000 7f8dd20a0640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:02.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.328+0000 7f8dd20a0640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:02.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.328+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8dcc0c6b00 con 0x7f8dcc152ab0 2026-02-22T21:40:02.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.331+0000 7f8da7fff640 1 -- 10.20.193.51:0/3994246047 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f8dc0066bf0 con 0x7f8dcc152ab0 2026-02-22T21:40:02.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.331+0000 7f8da7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:02.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.331+0000 7f8da7fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:02.333 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:02.333 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:02.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 10 librados: watch_flush enter 2026-02-22T21:40:02.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 10 librados: watch_flush exit 2026-02-22T21:40:02.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 20 client.4881.objecter shutdown clearing up homeless session... 2026-02-22T21:40:02.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 10 client.4881.objecter successfully canceled tick 2026-02-22T21:40:02.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 msgr2=0x7f8d9c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 0x7f8d9c07b310 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8db8002ab0 tx=0x7f8db8002a40 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 10 monclient: shutdown 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 msgr2=0x7f8dcc0c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000cb40 tx=0x7f8dc000a430 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 shutdown_connections 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8d9c078e50 0x7f8d9c07b310 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dcc152ab0 0x7f8dcc0c10d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dcc0c93d0 0x7f8dcc1f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 --2- 10.20.193.51:0/3994246047 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dcc06d9d0 0x7f8dcc1f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:02.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 >> 10.20.193.51:0/3994246047 conn(0x7f8dcc0cd7b0 msgr2=0x7f8dcc0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:02.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 shutdown_connections 2026-02-22T21:40:02.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 -- 10.20.193.51:0/3994246047 wait complete. 2026-02-22T21:40:02.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:02.332+0000 7f8dd20a0640 1 librados: shutdown 2026-02-22T21:40:05.342 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:05.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.412+0000 7ff6de12c640 10 monclient: get_monmap_and_config 2026-02-22T21:40:05.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.412+0000 7ff6de12c640 10 monclient: build_initial_monmap 2026-02-22T21:40:05.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.412+0000 7ff6de12c640 10 monclient: monmap: 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:05.413372+0000 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:05.413372+0000 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:05.433 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 Processor -- start 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 -- start start 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient: init 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient: _add_conns ranks=[3,5,4,0,1,2] 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff6d806ff10 0x7ff6d80702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d806ff10 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:05.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff6d815a0f0 0x7ff6d815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d815a0f0 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ff6d8153750 0x7ff6d8153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d8153750 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d815d150 con 0x7ff6d806ff10 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d815aa60 con 0x7ff6d8153750 2026-02-22T21:40:05.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d815d3f0 con 0x7ff6d815a0f0 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.413+0000 7ff6de12c640 10 monclient(hunting): authenticate will time out at 1280.993929s 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff6d806ff10 0x7ff6d80702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d7fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ff6d8153750 0x7ff6d8153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff6d806ff10 0x7ff6d80702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:53894/0 (socket says 10.20.193.51:53894) 2026-02-22T21:40:05.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 1 -- 10.20.193.51:0/2486269566 learned_addr learned my addr 10.20.193.51:0/2486269566 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): get_auth_request con 0x7ff6d806ff10 auth_method 0 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d7fff640 10 monclient(hunting): get_auth_request con 0x7ff6d8153750 auth_method 0 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d6ffd640 1 --2- 10.20.193.51:0/2486269566 >> v2:10.20.193.125:3300/0 conn(0x7ff6d815a0f0 0x7ff6d815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d6ffd640 10 monclient(hunting): get_auth_request con 0x7ff6d815a0f0 auth_method 0 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.414+0000 7ff6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_done global_id 4884 payload 274 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 -- 10.20.193.51:0/2486269566 >> v2:10.20.193.125:3300/0 conn(0x7ff6d815a0f0 msgr2=0x7ff6d815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 --2- 10.20.193.51:0/2486269566 >> v2:10.20.193.125:3300/0 conn(0x7ff6d815a0f0 0x7ff6d815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 -- 10.20.193.51:0/2486269566 >> v2:10.20.193.68:3300/0 conn(0x7ff6d8153750 msgr2=0x7ff6d8153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 --2- 10.20.193.51:0/2486269566 >> v2:10.20.193.68:3300/0 conn(0x7ff6d8153750 0x7ff6d8153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 monclient: found mon.noname-a 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.415+0000 7ff6d77fe640 1 -- 10.20.193.51:0/2486269566 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d815d6d0 con 0x7ff6d806ff10 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.416+0000 7ff6d77fe640 1 --2- 10.20.193.51:0/2486269566 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d806ff10 0x7ff6d80702f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80c62e0 tx=0x7ff6c402ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d86b5c4382dc8b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 1 -- 10.20.193.51:0/2486269566 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c403c070 con 0x7ff6d806ff10 2026-02-22T21:40:05.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: dump: 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.442 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 1 -- 10.20.193.51:0/2486269566 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c402fbd0 con 0x7ff6d806ff10 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6de12c640 5 monclient: authenticate success, global_id 4884 2026-02-22T21:40:05.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6de12c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 1 -- 10.20.193.51:0/2486269566 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4037440 con 0x7ff6d806ff10 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6de12c640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6de12c640 4 set_mon_vals no callback set 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.418+0000 7ff6d67fc640 10 monclient: dump: 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:05.444 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 10 monclient: shutdown 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:05.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 -- 10.20.193.51:0/2486269566 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d806ff10 msgr2=0x7ff6d80702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2486269566 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d806ff10 0x7ff6d80702f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80c62e0 tx=0x7ff6c402ef50 comp rx=0 tx=0).stop 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 -- 10.20.193.51:0/2486269566 shutdown_connections 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2486269566 >> v2:10.20.193.68:3300/0 conn(0x7ff6d8153750 0x7ff6d8153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2486269566 >> v2:10.20.193.125:3300/0 conn(0x7ff6d815a0f0 0x7ff6d815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2486269566 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d806ff10 0x7ff6d80702f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 -- 10.20.193.51:0/2486269566 >> 10.20.193.51:0/2486269566 conn(0x7ff6d80cd7b0 msgr2=0x7ff6d80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 -- 10.20.193.51:0/2486269566 shutdown_connections 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 -- 10.20.193.51:0/2486269566 wait complete. 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 10 monclient: build_initial_monmap 2026-02-22T21:40:05.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 10 monclient: monmap: 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 librados: starting msgr at 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 librados: starting objecter 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.419+0000 7ff6de12c640 1 Processor -- start 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 -- start start 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 librados: setting wanted keys 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 librados: calling monclient init 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient: init 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff6d806ff10 0x7ff6d806ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.448 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d806ff10 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff6d8153750 0x7ff6d806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d8153750 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d815a0f0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d815df70 con 0x7ff6d815a0f0 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d815dc90 con 0x7ff6d806ff10 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8159bd0 con 0x7ff6d8153750 2026-02-22T21:40:05.449 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6de12c640 10 monclient(hunting): authenticate will time out at 1281.000430s 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d77fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff6d806ff10 0x7ff6d806ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:53896/0 (socket says 10.20.193.51:53896) 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 1 -- 10.20.193.51:0/2511432496 learned_addr learned my addr 10.20.193.51:0/2511432496 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 10 monclient(hunting): get_auth_request con 0x7ff6d815a0f0 auth_method 0 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d6ffd640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff6d8153750 0x7ff6d806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d77fe640 10 monclient(hunting): get_auth_request con 0x7ff6d806ff10 auth_method 0 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.420+0000 7ff6d77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): get_auth_request con 0x7ff6d8153750 auth_method 0 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.421+0000 7ff6d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 10 monclient(hunting): handle_auth_done global_id 4887 payload 824 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 -- 10.20.193.51:0/2511432496 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff6d8153750 msgr2=0x7ff6d806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff6d8153750 0x7ff6d806d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 -- 10.20.193.51:0/2511432496 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff6d806ff10 msgr2=0x7ff6d806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff6d806ff10 0x7ff6d806ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 monclient: found mon.noname-a 2026-02-22T21:40:05.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.422+0000 7ff6d7fff640 1 -- 10.20.193.51:0/2511432496 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d81f19b0 con 0x7ff6d815a0f0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.423+0000 7ff6d7fff640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00eed0 tx=0x7ff6cc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.423+0000 7ff6b3fff640 10 client.?.objecter ms_handle_connect 0x7ff6d815a0f0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.423+0000 7ff6b3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc018020 con 0x7ff6d815a0f0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: dump: 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:05.453 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc014400 con 0x7ff6d815a0f0 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 5 monclient: authenticate success, global_id 4887 2026-02-22T21:40:05.454 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _renew_subs 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d81f1c90 con 0x7ff6d815a0f0 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0147f0 con 0x7ff6d815a0f0 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6d5ffb640 4 set_mon_vals no callback set 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6b3fff640 10 monclient: dump: 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.455 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 client.4887.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _renew_subs 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d81f2250 con 0x7ff6d815a0f0 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 1 librados: init done 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:05.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:05.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.425+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d80c9180 con 0x7ff6d815a0f0 2026-02-22T21:40:05.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.529+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff6cc014990 con 0x7ff6d815a0f0 2026-02-22T21:40:05.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.530+0000 7ff6b3fff640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 0x7ff6a807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:05.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.530+0000 7ff6d77fe640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 0x7ff6a807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:05.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.530+0000 7ff6d77fe640 10 monclient: get_auth_request con 0x7ff6a8078e50 auth_method 0 2026-02-22T21:40:05.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.530+0000 7ff6b3fff640 10 client.4887.objecter ms_handle_connect 0x7ff6a8078e50 2026-02-22T21:40:05.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.530+0000 7ff6d77fe640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 0x7ff6a807b310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80caee0 tx=0x7ff6c4039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:05.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff6cc05e9e0 con 0x7ff6d815a0f0 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 10 client.4887.objecter ms_dispatch2 0x7ff6d8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 3 client.4887.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 3 client.4887.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 20 client.4887.objecter dump_active .. 0 homeless 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff6cc022070 con 0x7ff6d815a0f0 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:05.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.537+0000 7ff6b3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:05.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.640+0000 7ff6de12c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:05.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.640+0000 7ff6de12c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:05.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.640+0000 7ff6de12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:05.642 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.640+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff6d80c8630 con 0x7ff6d815a0f0 2026-02-22T21:40:05.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.642+0000 7ff6b3fff640 1 -- 10.20.193.51:0/2511432496 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff6cc0629a0 con 0x7ff6d815a0f0 2026-02-22T21:40:05.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.642+0000 7ff6b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:05.644 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.642+0000 7ff6b3fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:05.644 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:05.644 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 10 librados: watch_flush enter 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 10 librados: watch_flush exit 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 20 client.4887.objecter shutdown clearing up homeless session... 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 10 client.4887.objecter successfully canceled tick 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 msgr2=0x7ff6a807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:05.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 0x7ff6a807b310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80caee0 tx=0x7ff6c4039040 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 10 monclient: shutdown 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 msgr2=0x7ff6d806dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00eed0 tx=0x7ff6cc00b480 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 shutdown_connections 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6a8078e50 0x7ff6a807b310 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff6d815a0f0 0x7ff6d806dee0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff6d8153750 0x7ff6d806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 --2- 10.20.193.51:0/2511432496 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff6d806ff10 0x7ff6d806ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 >> 10.20.193.51:0/2511432496 conn(0x7ff6d80cd7b0 msgr2=0x7ff6d80cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 shutdown_connections 2026-02-22T21:40:05.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 -- 10.20.193.51:0/2511432496 wait complete. 2026-02-22T21:40:05.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:05.643+0000 7ff6de12c640 1 librados: shutdown 2026-02-22T21:40:08.655 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:08.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.721+0000 7f86e4b98640 10 monclient: get_monmap_and_config 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.721+0000 7f86e4b98640 10 monclient: build_initial_monmap 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.721+0000 7f86e4b98640 10 monclient: monmap: 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:08.722451+0000 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:08.722451+0000 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:08.724 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 Processor -- start 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 -- start start 2026-02-22T21:40:08.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient: init 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient: _add_conns ranks=[3,1,5,0,4,2] 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f86e0150220 0x7f86e00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-a con 0x7f86e0150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f86e0150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:08.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f86e015c710 0x7f86e006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-c con 0x7f86e015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e015a920 con 0x7f86e0150600 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f86e015af60 con 0x7f86e0150220 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f86e015b240 con 0x7f86e015c710 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.722+0000 7f86e4b98640 10 monclient(hunting): authenticate will time out at 1284.302800s 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f86e0150220 0x7f86e00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ddd74640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f86e0150600 0x7f86e015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:57874/0 (socket says 10.20.193.51:57874) 2026-02-22T21:40:08.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ddd74640 1 -- 10.20.193.51:0/3843107194 learned_addr learned my addr 10.20.193.51:0/3843107194 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): get_auth_request con 0x7f86e0150220 auth_method 0 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 1 --2- 10.20.193.51:0/3843107194 >> v2:10.20.193.125:3300/0 conn(0x7f86e015c710 0x7f86e006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): get_auth_request con 0x7f86e015c710 auth_method 0 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.723+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 1 -- 10.20.193.51:0/3843107194 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563505316 0 0) 0x7f86e015a920 con 0x7f86e0150600 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 10 monclient(hunting): _negotiate 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 10 monclient(hunting): my global_id is 4864 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86dd573640 1 -- 10.20.193.51:0/3843107194 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86cc003620 con 0x7f86e0150600 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 10 monclient(hunting): handle_auth_done global_id 4890 payload 274 2026-02-22T21:40:08.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 1 -- 10.20.193.51:0/3843107194 >> v2:10.20.193.125:3300/0 conn(0x7f86e015c710 msgr2=0x7f86e006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 1 --2- 10.20.193.51:0/3843107194 >> v2:10.20.193.125:3300/0 conn(0x7f86e015c710 0x7f86e006fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 1 -- 10.20.193.51:0/3843107194 >> v1:10.20.193.68:6789/0 conn(0x7f86e0150600 legacy=0x7f86e015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 10 monclient: _finish_hunting 0 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 1 monclient: found mon.noname-a 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.724+0000 7f86de575640 1 -- 10.20.193.51:0/3843107194 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e015b520 con 0x7f86e0150220 2026-02-22T21:40:08.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.726+0000 7f86de575640 1 --2- 10.20.193.51:0/3843107194 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e0150220 0x7f86e00c73a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f86e00c65c0 tx=0x7f86d402eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5d07322d9505f9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:08.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 1 -- 10.20.193.51:0/3843107194 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d403c070 con 0x7f86e0150220 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: dump: 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.740 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: _finish_auth 0 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: _check_auth_tickets 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 1 -- 10.20.193.51:0/3843107194 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d402fb80 con 0x7f86e0150220 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 1 -- 10.20.193.51:0/3843107194 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4037440 con 0x7f86e0150220 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 5 monclient: authenticate success, global_id 4890 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:08.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 4 set_mon_vals no callback set 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86dd573640 10 monclient: dump: 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:08.742 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 10 monclient: shutdown 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 -- 10.20.193.51:0/3843107194 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e0150220 msgr2=0x7f86e00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 --2- 10.20.193.51:0/3843107194 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e0150220 0x7f86e00c73a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f86e00c65c0 tx=0x7f86d402eeb0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 -- 10.20.193.51:0/3843107194 shutdown_connections 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 --2- 10.20.193.51:0/3843107194 >> v2:10.20.193.125:3300/0 conn(0x7f86e015c710 0x7f86e006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 --2- 10.20.193.51:0/3843107194 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e0150220 0x7f86e00c73a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 -- 10.20.193.51:0/3843107194 >> 10.20.193.51:0/3843107194 conn(0x7f86e00cd7b0 msgr2=0x7f86e00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 -- 10.20.193.51:0/3843107194 shutdown_connections 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.728+0000 7f86e4b98640 1 -- 10.20.193.51:0/3843107194 wait complete. 2026-02-22T21:40:08.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 10 monclient: build_initial_monmap 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 10 monclient: monmap: 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 librados: starting msgr at 2026-02-22T21:40:08.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 librados: starting objecter 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 Processor -- start 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 -- start start 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 librados: setting wanted keys 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 1 librados: calling monclient init 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.729+0000 7f86e4b98640 10 monclient: init 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f86e0150220 0x7f86e0072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-b con 0x7f86e0150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f86e0150600 0x7f86e00bf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-c con 0x7f86e0150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:08.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): picked mon.noname-a con 0x7f86e015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f86e00701e0 con 0x7f86e015c710 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f86e015bae0 con 0x7f86e0150220 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f86e0070360 con 0x7f86e0150600 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86e4b98640 10 monclient(hunting): authenticate will time out at 1284.310384s 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ddd74640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f86e0150600 0x7f86e00bf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86de575640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f86e0150220 0x7f86e0072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:44974/0 (socket says 10.20.193.51:44974) 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 1 -- 10.20.193.51:0/2712664432 learned_addr learned my addr 10.20.193.51:0/2712664432 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 10 monclient(hunting): get_auth_request con 0x7f86e015c710 auth_method 0 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ded76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ddd74640 10 monclient(hunting): get_auth_request con 0x7f86e0150600 auth_method 0 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ddd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ddd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.730+0000 7f86ddd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): get_auth_request con 0x7f86e0150220 auth_method 0 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:08.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ded76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86de575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ddd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ddd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.731+0000 7f86ddd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 -- 10.20.193.51:0/2712664432 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f86e0150600 msgr2=0x7f86e00bf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f86e0150600 0x7f86e00bf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 -- 10.20.193.51:0/2712664432 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f86e0150220 msgr2=0x7f86e0072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f86e0150220 0x7f86e0072910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 10 monclient: _finish_hunting 0 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 monclient: found mon.noname-a 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.732+0000 7f86ded76640 1 -- 10.20.193.51:0/2712664432 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e0074020 con 0x7f86e015c710 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.733+0000 7f86ded76640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f86d000efc0 tx=0x7f86d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.733+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86e015c710 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.733+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0018020 con 0x7f86e015c710 2026-02-22T21:40:08.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: dump: 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:08.750 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 5 monclient: authenticate success, global_id 4893 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d0014480 con 0x7f86e015c710 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0014870 con 0x7f86e015c710 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _renew_subs 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86dcd72640 4 set_mon_vals no callback set 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e0074300 con 0x7f86e015c710 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:08.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86c6ffd640 10 monclient: dump: 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:08.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _renew_subs 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e00bed20 con 0x7f86e015c710 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 1 librados: init done 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.735+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e006fca0 con 0x7f86e015c710 2026-02-22T21:40:08.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f86d0014a10 con 0x7f86e015c710 2026-02-22T21:40:08.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86c6ffd640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 0x7f86ac07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:08.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86de575640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 0x7f86ac07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:08.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86de575640 10 monclient: get_auth_request con 0x7f86ac078e80 auth_method 0 2026-02-22T21:40:08.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86c6ffd640 10 client.4893.objecter ms_handle_connect 0x7f86ac078e80 2026-02-22T21:40:08.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.842+0000 7f86de575640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 0x7f86ac07b340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f86d4039000 tx=0x7f86d4033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f86d005ed80 con 0x7f86e015c710 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 10 client.4893.objecter ms_dispatch2 0x7f86e0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 3 client.4893.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 3 client.4893.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 20 client.4893.objecter dump_active .. 0 homeless 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f86d0022070 con 0x7f86e015c710 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:08.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.850+0000 7f86c6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:08.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.950+0000 7f86e4b98640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:08.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.950+0000 7f86e4b98640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:08.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.950+0000 7f86e4b98640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:08.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.950+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f86e0074da0 con 0x7f86e015c710 2026-02-22T21:40:08.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.953+0000 7f86c6ffd640 1 -- 10.20.193.51:0/2712664432 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f86d0010040 con 0x7f86e015c710 2026-02-22T21:40:08.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.953+0000 7f86c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:08.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.953+0000 7f86c6ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:08.955 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:08.955 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 10 librados: watch_flush enter 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 10 librados: watch_flush exit 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 20 client.4893.objecter shutdown clearing up homeless session... 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 10 client.4893.objecter successfully canceled tick 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 msgr2=0x7f86ac07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 0x7f86ac07b340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f86d4039000 tx=0x7f86d4033000 comp rx=0 tx=0).stop 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 10 monclient: shutdown 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 msgr2=0x7f86e00bfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f86d000efc0 tx=0x7f86d000c490 comp rx=0 tx=0).stop 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 shutdown_connections 2026-02-22T21:40:08.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f86ac078e80 0x7f86ac07b340 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f86e015c710 0x7f86e00bfa60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f86e0150600 0x7f86e00bf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 --2- 10.20.193.51:0/2712664432 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f86e0150220 0x7f86e0072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 >> 10.20.193.51:0/2712664432 conn(0x7f86e00cd7b0 msgr2=0x7f86e006eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 shutdown_connections 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 -- 10.20.193.51:0/2712664432 wait complete. 2026-02-22T21:40:08.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:08.955+0000 7f86e4b98640 1 librados: shutdown 2026-02-22T21:40:11.965 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:12.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 10 monclient: get_monmap_and_config 2026-02-22T21:40:12.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 10 monclient: build_initial_monmap 2026-02-22T21:40:12.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 10 monclient: monmap: 2026-02-22T21:40:12.035 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:12.034052+0000 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:12.034052+0000 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:12.036 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:12.037 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 1 Processor -- start 2026-02-22T21:40:12.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 1 -- start start 2026-02-22T21:40:12.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.033+0000 7fb2cf3f7640 10 monclient: init 2026-02-22T21:40:12.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:12.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient: _add_conns ranks=[2,0,1,3,4,5] 2026-02-22T21:40:12.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2c8150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:12.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2c815c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb2c80703b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c815a920 con 0x7fb2c815c710 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c815af60 con 0x7fb2c80703b0 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c815b240 con 0x7fb2c8150220 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cf3f7640 10 monclient(hunting): authenticate will time out at 1287.614357s 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cc96b640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fb2c815c710 0x7fb2c806fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58988/0 (socket says 10.20.193.51:58988) 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.034+0000 7fb2cc96b640 1 -- 10.20.193.51:0/4093016910 learned_addr learned my addr 10.20.193.51:0/4093016910 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490463142 0 0) 0x7fb2c815a920 con 0x7fb2c815c710 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 10 monclient(hunting): _negotiate 2026-02-22T21:40:12.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 10 monclient(hunting): my global_id is 4896 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.035+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b0003620 con 0x7fb2c815c710 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1637716464 0 0) 0x7fb2c815b240 con 0x7fb2c8150220 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _negotiate 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): my global_id is 4838 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2c815a920 con 0x7fb2c8150220 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361904356 0 0) 0x7fb2c815af60 con 0x7fb2c80703b0 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _negotiate 2026-02-22T21:40:12.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 10 monclient(hunting): my global_id is 4870 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.036+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2c815b240 con 0x7fb2c80703b0 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 430542999 0 0) 0x7fb2c815a920 con 0x7fb2c8150220 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 >> v1:10.20.193.68:6789/0 conn(0x7fb2c80703b0 legacy=0x7fb2c8159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 >> v1:10.20.193.51:6789/0 conn(0x7fb2c815c710 legacy=0x7fb2c806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c815b520 con 0x7fb2c8150220 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:12.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fb2b8002ee0 con 0x7fb2c8150220 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2cf3f7640 5 monclient: authenticate success, global_id 4838 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2cf3f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: dump: 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.047 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 >> v1:10.20.193.125:6789/0 conn(0x7fb2c8150220 legacy=0x7fb2c815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2b0003fe0 0x7fb2b0004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): picked mon.c con 0x7fb2b0003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2b00052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): picked mon.a con 0x7fb2b0004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2b0009ab0 0x7fb2b000bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): picked mon.b con 0x7fb2b0009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b000c490 con 0x7fb2b0004e40 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b000c640 con 0x7fb2b0009ab0 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b000c840 con 0x7fb2b0003fe0 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:12.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2b7fff640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2cf3f7640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2cd96d640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2b0009ab0 0x7fb2b000bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.038+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2b00052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): get_auth_request con 0x7fb2b0004e40 auth_method 0 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): get_auth_request con 0x7fb2b0009ab0 auth_method 0 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:40:12.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd16c640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2b0003fe0 0x7fb2b0004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd16c640 10 monclient(hunting): get_auth_request con 0x7fb2b0003fe0 auth_method 0 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd16c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd16c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.039+0000 7fb2cd16c640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:40:12.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_done global_id 4838 payload 293 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 -- 10.20.193.51:0/4093016910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2b0003fe0 msgr2=0x7fb2b0004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2b0003fe0 0x7fb2b0004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 -- 10.20.193.51:0/4093016910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2b0009ab0 msgr2=0x7fb2b000bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2b0009ab0 0x7fb2b000bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 10 monclient: _finish_hunting 0 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 monclient: found mon.a 2026-02-22T21:40:12.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cd96d640 1 -- 10.20.193.51:0/4093016910 reap_dead start 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 1 -- 10.20.193.51:0/4093016910 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c815a920 con 0x7fb2b0004e40 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 10 monclient: _finish_auth 0 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 10 monclient: _check_auth_tickets 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.040+0000 7fb2cc96b640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:12.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.041+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2b00052c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc003370 tx=0x7fb2bc02f060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6600a80ad22fa368 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:12.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc002f90 con 0x7fb2b0004e40 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: dump: 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.060 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc033070 con 0x7fb2b0004e40 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 1 -- 10.20.193.51:0/4093016910 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc004de0 con 0x7fb2b0004e40 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2cf3f7640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2cf3f7640 4 set_mon_vals no callback set 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:12.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.042+0000 7fb2b7fff640 10 monclient: dump: 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 10 monclient: shutdown 2026-02-22T21:40:12.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 msgr2=0x7fb2b00052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2b00052c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc003370 tx=0x7fb2bc02f060 comp rx=0 tx=0).stop 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/4093016910 shutdown_connections 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/4093016910 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2b00052c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/4093016910 >> 10.20.193.51:0/4093016910 conn(0x7fb2c80cd7b0 msgr2=0x7fb2c80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/4093016910 shutdown_connections 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.043+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/4093016910 wait complete. 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient: build_initial_monmap 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient: monmap: 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:12.063 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 librados: starting msgr at 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 librados: starting objecter 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 Processor -- start 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 -- start start 2026-02-22T21:40:12.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 librados: setting wanted keys 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 librados: calling monclient init 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient: init 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2c80c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-a con 0x7fb2b0004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 0x7fb2c80c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c81e8680 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 0x7fb2c81f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c80c98e0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8150600 con 0x7fb2b0004e40 2026-02-22T21:40:12.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c815bae0 con 0x7fb2c80c98e0 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8150780 con 0x7fb2c81e8680 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.044+0000 7fb2cf3f7640 10 monclient(hunting): authenticate will time out at 1287.625004s 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 0x7fb2c81f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 0x7fb2c80c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd16c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2c80c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 0x7fb2c81f1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:57036/0 (socket says 10.20.193.51:57036) 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 1 -- 10.20.193.51:0/1639503598 learned_addr learned my addr 10.20.193.51:0/1639503598 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): get_auth_request con 0x7fb2c80c98e0 auth_method 0 2026-02-22T21:40:12.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): get_auth_request con 0x7fb2c81e8680 auth_method 0 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd16c640 10 monclient(hunting): get_auth_request con 0x7fb2b0004e40 auth_method 0 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd16c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd16c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd16c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.067 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cd96d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.045+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.046+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.046+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.046+0000 7fb2cd16c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 -- 10.20.193.51:0/1639503598 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 msgr2=0x7fb2c81f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 0x7fb2c81f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 -- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 msgr2=0x7fb2c80c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2c80c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.068 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 10 monclient: _finish_hunting 0 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 monclient: found mon.noname-c 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.047+0000 7fb2cc96b640 1 -- 10.20.193.51:0/1639503598 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c814d330 con 0x7fb2c81e8680 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.048+0000 7fb2cc96b640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 0x7fb2c80c93a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc005000 tx=0x7fb2bc002b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.048+0000 7fb2b5ffb640 10 client.?.objecter ms_handle_connect 0x7fb2c81e8680 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.048+0000 7fb2b5ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc007c60 con 0x7fb2c81e8680 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: dump: 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:12.069 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: _finish_auth 0 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 5 monclient: authenticate success, global_id 4841 2026-02-22T21:40:12.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _renew_subs 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c81f5000 con 0x7fb2c81e8680 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 client.4841.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _renew_subs 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81f55c0 con 0x7fb2c81e8680 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 1 librados: init done 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c80c73a0 con 0x7fb2c81e8680 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc033070 con 0x7fb2c81e8680 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:12.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc049060 con 0x7fb2c81e8680 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b77fe640 4 set_mon_vals no callback set 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.049+0000 7fb2b5ffb640 10 monclient: dump: 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:12.072 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:12.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.157+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb2bc02fcd0 con 0x7fb2c81e8680 2026-02-22T21:40:12.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.158+0000 7fb2b5ffb640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 0x7fb29807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:12.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.158+0000 7fb2cd16c640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 0x7fb29807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:12.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.158+0000 7fb2cd16c640 10 monclient: get_auth_request con 0x7fb298078e00 auth_method 0 2026-02-22T21:40:12.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.158+0000 7fb2b5ffb640 10 client.4841.objecter ms_handle_connect 0x7fb298078e00 2026-02-22T21:40:12.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.158+0000 7fb2cd16c640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 0x7fb29807b2c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8002970 tx=0x7fb2b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:12.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.159+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb2bc081be0 con 0x7fb2c81e8680 2026-02-22T21:40:12.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.159+0000 7fb2b5ffb640 10 client.4841.objecter ms_dispatch2 0x7fb2c8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:12.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.159+0000 7fb2b5ffb640 3 client.4841.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:12.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.159+0000 7fb2b5ffb640 3 client.4841.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:12.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.159+0000 7fb2b5ffb640 20 client.4841.objecter dump_active .. 0 homeless 2026-02-22T21:40:12.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.166+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb2bc085ba0 con 0x7fb2c81e8680 2026-02-22T21:40:12.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.166+0000 7fb2b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:12.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.166+0000 7fb2b5ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:12.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.264+0000 7fb2cf3f7640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:12.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.264+0000 7fb2cf3f7640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:12.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.264+0000 7fb2cf3f7640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:12.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.264+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2c8074da0 con 0x7fb2c81e8680 2026-02-22T21:40:12.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.267+0000 7fb2b5ffb640 1 -- 10.20.193.51:0/1639503598 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb2bc089ea0 con 0x7fb2c81e8680 2026-02-22T21:40:12.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.267+0000 7fb2b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:12.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.267+0000 7fb2b5ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:12.268 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:12.268 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 10 librados: watch_flush enter 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 10 librados: watch_flush exit 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 20 client.4841.objecter shutdown clearing up homeless session... 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 10 client.4841.objecter successfully canceled tick 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 msgr2=0x7fb29807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 0x7fb29807b2c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8002970 tx=0x7fb2b8009040 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 10 monclient: shutdown 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 msgr2=0x7fb2c80c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 0x7fb2c80c93a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc005000 tx=0x7fb2bc002b20 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 shutdown_connections 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb298078e00 0x7fb29807b2c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb2c80c98e0 0x7fb2c81f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb2c81e8680 0x7fb2c80c93a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 --2- 10.20.193.51:0/1639503598 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb2b0004e40 0x7fb2c80c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 >> 10.20.193.51:0/1639503598 conn(0x7fb2c80cd7b0 msgr2=0x7fb2c814da40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 shutdown_connections 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 -- 10.20.193.51:0/1639503598 wait complete. 2026-02-22T21:40:12.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:12.268+0000 7fb2cf3f7640 1 librados: shutdown 2026-02-22T21:40:15.278 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:15.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 10 monclient: get_monmap_and_config 2026-02-22T21:40:15.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 10 monclient: build_initial_monmap 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 10 monclient: monmap: 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:15.347119+0000 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:15.347119+0000 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 1 Processor -- start 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 1 -- start start 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.346+0000 7fd8c583c640 10 monclient: init 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient: _add_conns ranks=[2,5,4,0,3,1] 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8c0150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd8c015c710 0x7fd8c00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd8c0150600 0x7fd8c006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c0150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c015a920 con 0x7fd8c0150220 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c015af60 con 0x7fd8c0150600 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c015b240 con 0x7fd8c015c710 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:15.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8c583c640 10 monclient(hunting): authenticate will time out at 1290.927503s 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.347+0000 7fd8bf7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd8c0150600 0x7fd8c006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8be7fc640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd8c015c710 0x7fd8c00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd8c0150600 0x7fd8c006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:57042/0 (socket says 10.20.193.51:57042) 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 1 -- 10.20.193.51:0/3415223611 learned_addr learned my addr 10.20.193.51:0/3415223611 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd8c0150600 auth_method 0 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8be7fc640 10 monclient(hunting): get_auth_request con 0x7fd8c015c710 auth_method 0 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8be7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.348+0000 7fd8be7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:15.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_done global_id 4876 payload 274 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 1 -- 10.20.193.51:0/3415223611 >> v2:10.20.193.125:3300/0 conn(0x7fd8c015c710 msgr2=0x7fd8c00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 1 --2- 10.20.193.51:0/3415223611 >> v2:10.20.193.125:3300/0 conn(0x7fd8c015c710 0x7fd8c00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 1 -- 10.20.193.51:0/3415223611 >> v1:10.20.193.125:6789/0 conn(0x7fd8c0150220 legacy=0x7fd8c015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 1 monclient: found mon.noname-b 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.349+0000 7fd8bf7fe640 1 -- 10.20.193.51:0/3415223611 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c015b520 con 0x7fd8c0150600 2026-02-22T21:40:15.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.350+0000 7fd8bf7fe640 1 --2- 10.20.193.51:0/3415223611 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150600 0x7fd8c006fca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009e50 tx=0x7fd8b002f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386057db18cf336e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 1 -- 10.20.193.51:0/3415223611 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b002fdf0 con 0x7fd8c0150600 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: dump: 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:15.353 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: _finish_auth 0 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: _check_auth_tickets 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 1 -- 10.20.193.51:0/3415223611 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b00373f0 con 0x7fd8c0150600 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 5 monclient: authenticate success, global_id 4876 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 1 -- 10.20.193.51:0/3415223611 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00377c0 con 0x7fd8c0150600 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8bdffb640 10 monclient: dump: 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:15.354 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 4 set_mon_vals no callback set 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 10 monclient: shutdown 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 -- 10.20.193.51:0/3415223611 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150600 msgr2=0x7fd8c006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3415223611 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150600 0x7fd8c006fca0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009e50 tx=0x7fd8b002f330 comp rx=0 tx=0).stop 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 -- 10.20.193.51:0/3415223611 shutdown_connections 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3415223611 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150600 0x7fd8c006fca0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3415223611 >> v2:10.20.193.125:3300/0 conn(0x7fd8c015c710 0x7fd8c00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 -- 10.20.193.51:0/3415223611 >> 10.20.193.51:0/3415223611 conn(0x7fd8c00cd7b0 msgr2=0x7fd8c00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 -- 10.20.193.51:0/3415223611 shutdown_connections 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.352+0000 7fd8c583c640 1 -- 10.20.193.51:0/3415223611 wait complete. 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 10 monclient: build_initial_monmap 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 10 monclient: monmap: 2026-02-22T21:40:15.356 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 librados: starting msgr at 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 librados: starting objecter 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 Processor -- start 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 -- start start 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 librados: setting wanted keys 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 1 librados: calling monclient init 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.353+0000 7fd8c583c640 10 monclient: init 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c0150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd8c0150600 0x7fd8c01e5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c0150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd8c015c710 0x7fd8c01e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c00701e0 con 0x7fd8c015c710 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c015bae0 con 0x7fd8c0150220 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0070360 con 0x7fd8c0150600 2026-02-22T21:40:15.357 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8c583c640 10 monclient(hunting): authenticate will time out at 1290.934209s 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd8c0150600 0x7fd8c01e5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8bf7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd8c015c710 0x7fd8c01e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:57050/0 (socket says 10.20.193.51:57050) 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 1 -- 10.20.193.51:0/3257396077 learned_addr learned my addr 10.20.193.51:0/3257396077 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): get_auth_request con 0x7fd8c0150220 auth_method 0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): get_auth_request con 0x7fd8c0150600 auth_method 0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8bf7fe640 10 monclient(hunting): get_auth_request con 0x7fd8c015c710 auth_method 0 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8bf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.354+0000 7fd8be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.355+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.355+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:15.358 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.355+0000 7fd8bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 10 monclient(hunting): handle_auth_done global_id 4879 payload 824 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 -- 10.20.193.51:0/3257396077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd8c0150600 msgr2=0x7fd8c01e5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd8c0150600 0x7fd8c01e5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 -- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd8c015c710 msgr2=0x7fd8c01e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd8c015c710 0x7fd8c01e58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 10 monclient: _finish_hunting 0 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 monclient: found mon.noname-b 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.356+0000 7fd8beffd640 1 -- 10.20.193.51:0/3257396077 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c00c16f0 con 0x7fd8c0150220 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.357+0000 7fd8beffd640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c700 tx=0x7fd8b400cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.357+0000 7fd897fff640 10 client.?.objecter ms_handle_connect 0x7fd8c0150220 2026-02-22T21:40:15.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.357+0000 7fd897fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:15.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40052d0 con 0x7fd8c0150220 2026-02-22T21:40:15.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:15.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:15.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: dump: 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b4005470 con 0x7fd8c0150220 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40038d0 con 0x7fd8c0150220 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8bd7fa640 4 set_mon_vals no callback set 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd897fff640 10 monclient: dump: 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:15.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 5 monclient: authenticate success, global_id 4879 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _renew_subs 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c00c3810 con 0x7fd8c0150220 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 client.4879.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _renew_subs 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c00c3dd0 con 0x7fd8c0150220 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 1 librados: init done 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.359+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c00c0d00 con 0x7fd8c0150220 2026-02-22T21:40:15.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.371+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd8b4003a70 con 0x7fd8c0150220 2026-02-22T21:40:15.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.372+0000 7fd897fff640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 0x7fd89007b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:15.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.372+0000 7fd8be7fc640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 0x7fd89007b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:15.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.372+0000 7fd8be7fc640 10 monclient: get_auth_request con 0x7fd890078e30 auth_method 0 2026-02-22T21:40:15.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.372+0000 7fd897fff640 10 client.4879.objecter ms_handle_connect 0x7fd890078e30 2026-02-22T21:40:15.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.372+0000 7fd8be7fc640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 0x7fd89007b2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd8c01e7470 tx=0x7fd8a8009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:15.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.373+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd8b405e020 con 0x7fd8c0150220 2026-02-22T21:40:15.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.373+0000 7fd897fff640 10 client.4879.objecter ms_dispatch2 0x7fd8c0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:15.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.373+0000 7fd897fff640 3 client.4879.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:15.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.373+0000 7fd897fff640 3 client.4879.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:15.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.373+0000 7fd897fff640 20 client.4879.objecter dump_active .. 0 homeless 2026-02-22T21:40:15.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.475+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd8c00c0d00 con 0x7fd8c0150220 2026-02-22T21:40:15.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.475+0000 7fd897fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:15.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.475+0000 7fd897fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:15.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.575+0000 7fd8c583c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:15.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.575+0000 7fd8c583c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:15.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.575+0000 7fd8c583c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:15.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.575+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd8c015e110 con 0x7fd8c0150220 2026-02-22T21:40:15.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.578+0000 7fd897fff640 1 -- 10.20.193.51:0/3257396077 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fd8c015e110 con 0x7fd8c0150220 2026-02-22T21:40:15.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.578+0000 7fd897fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:15.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.578+0000 7fd897fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:15.580 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:15.580 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 10 librados: watch_flush enter 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 10 librados: watch_flush exit 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 20 client.4879.objecter shutdown clearing up homeless session... 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 10 client.4879.objecter successfully canceled tick 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 msgr2=0x7fd89007b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 0x7fd89007b2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd8c01e7470 tx=0x7fd8a8009290 comp rx=0 tx=0).stop 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 10 monclient: shutdown 2026-02-22T21:40:15.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 msgr2=0x7fd8c00c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c700 tx=0x7fd8b400cbd0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 shutdown_connections 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd890078e30 0x7fd89007b2f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd8c015c710 0x7fd8c01e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd8c0150600 0x7fd8c01e5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 --2- 10.20.193.51:0/3257396077 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd8c0150220 0x7fd8c00c10d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 >> 10.20.193.51:0/3257396077 conn(0x7fd8c00cd7b0 msgr2=0x7fd8c006eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 shutdown_connections 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 -- 10.20.193.51:0/3257396077 wait complete. 2026-02-22T21:40:15.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:15.579+0000 7fd8c583c640 1 librados: shutdown 2026-02-22T21:40:18.590 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.655+0000 7f9d830b2640 10 monclient: get_monmap_and_config 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.655+0000 7f9d830b2640 10 monclient: build_initial_monmap 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.655+0000 7f9d830b2640 10 monclient: monmap: 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:18.656314+0000 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:18.656314+0000 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:18.657 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 Processor -- start 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 -- start start 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient: init 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient: _add_conns ranks=[5,1,2,3,4,0] 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9d7c150220 0x7f9d7c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7c150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f9d7c150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:18.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d7c15c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7c15a920 con 0x7f9d7c150600 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7c15af60 con 0x7f9d7c15c710 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f9d7c15b240 con 0x7f9d7c150220 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.656+0000 7f9d830b2640 10 monclient(hunting): authenticate will time out at 1294.236606s 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d80e27640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9d7c150220 0x7f9d7c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d73fff640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f9d7c150600 0x7f9d7c15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:50490/0 (socket says 10.20.193.51:50490) 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d73fff640 1 -- 10.20.193.51:0/3474271924 learned_addr learned my addr 10.20.193.51:0/3474271924 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d80e27640 10 monclient(hunting): get_auth_request con 0x7f9d7c150220 auth_method 0 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d80e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d80e27640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:18.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.657+0000 7f9d80e27640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 1 -- 10.20.193.51:0/3474271924 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988242597 0 0) 0x7f9d7c15a920 con 0x7f9d7c150600 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 10 monclient(hunting): _negotiate 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 10 monclient(hunting): my global_id is 4882 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.658+0000 7f9d737fe640 1 -- 10.20.193.51:0/3474271924 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d68003620 con 0x7f9d7c150600 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 10 monclient(hunting): handle_auth_done global_id 4853 payload 274 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 1 -- 10.20.193.51:0/3474271924 >> v1:10.20.193.125:6789/0 conn(0x7f9d7c15c710 legacy=0x7f9d7c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 1 -- 10.20.193.51:0/3474271924 >> v1:10.20.193.68:6789/0 conn(0x7f9d7c150600 legacy=0x7f9d7c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 10 monclient: _finish_hunting 0 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 1 monclient: found mon.noname-c 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.660 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.659+0000 7f9d80e27640 1 -- 10.20.193.51:0/3474271924 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c15b520 con 0x7f9d7c150220 2026-02-22T21:40:18.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.661+0000 7f9d80e27640 1 --2- 10.20.193.51:0/3474271924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150220 0x7f9d7c0c73a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0098a0 tx=0x7f9d6c02ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e250386a7828131 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 1 -- 10.20.193.51:0/3474271924 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c03c070 con 0x7f9d7c150220 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: dump: 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: _finish_auth 0 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: _check_auth_tickets 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 1 -- 10.20.193.51:0/3474271924 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6c02fc10 con 0x7f9d7c150220 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 1 -- 10.20.193.51:0/3474271924 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c037440 con 0x7f9d7c150220 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 5 monclient: authenticate success, global_id 4853 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:18.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d737fe640 10 monclient: dump: 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 4 set_mon_vals no callback set 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 10 monclient: shutdown 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 -- 10.20.193.51:0/3474271924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150220 msgr2=0x7f9d7c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 --2- 10.20.193.51:0/3474271924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150220 0x7f9d7c0c73a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0098a0 tx=0x7f9d6c02ef00 comp rx=0 tx=0).stop 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 -- 10.20.193.51:0/3474271924 shutdown_connections 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 --2- 10.20.193.51:0/3474271924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150220 0x7f9d7c0c73a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 -- 10.20.193.51:0/3474271924 >> 10.20.193.51:0/3474271924 conn(0x7f9d7c0cd7b0 msgr2=0x7f9d7c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 -- 10.20.193.51:0/3474271924 shutdown_connections 2026-02-22T21:40:18.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 1 -- 10.20.193.51:0/3474271924 wait complete. 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 10 monclient: build_initial_monmap 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.663+0000 7f9d830b2640 10 monclient: monmap: 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:18.668 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 librados: starting msgr at 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 librados: starting objecter 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 Processor -- start 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 -- start start 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 librados: setting wanted keys 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 librados: calling monclient init 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient: init 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d7c150220 0x7f9d7c1f2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-c con 0x7f9d7c150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d7c15c710 0x7f9d7c0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): picked mon.noname-a con 0x7f9d7c15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c0703b0 con 0x7f9d7c15c710 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c15bae0 con 0x7f9d7c150220 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c070530 con 0x7f9d7c150600 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.664+0000 7f9d830b2640 10 monclient(hunting): authenticate will time out at 1294.244680s 2026-02-22T21:40:18.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d7c15c710 0x7f9d7c0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d7c150220 0x7f9d7c1f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36254/0 (socket says 10.20.193.51:36254) 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 1 -- 10.20.193.51:0/987658069 learned_addr learned my addr 10.20.193.51:0/987658069 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): get_auth_request con 0x7f9d7c150600 auth_method 0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): get_auth_request con 0x7f9d7c15c710 auth_method 0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): get_auth_request con 0x7f9d7c150220 auth_method 0 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d81628640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:18.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.665+0000 7f9d80e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 10 monclient(hunting): handle_auth_done global_id 4859 payload 824 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 -- 10.20.193.51:0/987658069 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d7c150220 msgr2=0x7f9d7c1f2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d7c150220 0x7f9d7c1f2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 -- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d7c15c710 msgr2=0x7f9d7c0c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d7c15c710 0x7f9d7c0c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 monclient: found mon.noname-c 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.666+0000 7f9d73fff640 1 -- 10.20.193.51:0/987658069 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c0c1bc0 con 0x7f9d7c150600 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.667+0000 7f9d73fff640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c4d0 tx=0x7f9d6400c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.667+0000 7f9d717fa640 10 client.?.objecter ms_handle_connect 0x7f9d7c150600 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.667+0000 7f9d717fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64017070 con 0x7f9d7c150600 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: dump: 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: _finish_auth 0 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: _check_auth_tickets 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 5 monclient: authenticate success, global_id 4859 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _renew_subs 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c0c1ea0 con 0x7f9d7c150600 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 client.4859.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _renew_subs 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c0c7770 con 0x7f9d7c150600 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 1 librados: init done 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d64005de0 con 0x7f9d7c150600 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64021060 con 0x7f9d7c150600 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d717fa640 10 monclient: dump: 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:18.673 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:18.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d72ffd640 4 set_mon_vals no callback set 2026-02-22T21:40:18.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:18.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:18.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.669+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7c0c73a0 con 0x7f9d7c150600 2026-02-22T21:40:18.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.680+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9d64024080 con 0x7f9d7c150600 2026-02-22T21:40:18.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.681+0000 7f9d717fa640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 0x7f9d4c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:18.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.681+0000 7f9d80e27640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 0x7f9d4c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:18.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.681+0000 7f9d80e27640 10 monclient: get_auth_request con 0x7f9d4c078e00 auth_method 0 2026-02-22T21:40:18.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.681+0000 7f9d717fa640 10 client.4859.objecter ms_handle_connect 0x7f9d4c078e00 2026-02-22T21:40:18.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.681+0000 7f9d80e27640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 0x7f9d4c07b2c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c039000 tx=0x7f9d6c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:18.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.683+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9d64096ce0 con 0x7f9d7c150600 2026-02-22T21:40:18.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.683+0000 7f9d717fa640 10 client.4859.objecter ms_dispatch2 0x7f9d7c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:18.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.683+0000 7f9d717fa640 3 client.4859.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:18.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.683+0000 7f9d717fa640 3 client.4859.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:18.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.683+0000 7f9d717fa640 20 client.4859.objecter dump_active .. 0 homeless 2026-02-22T21:40:18.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.787+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9d640614b0 con 0x7f9d7c150600 2026-02-22T21:40:18.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.787+0000 7f9d717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:18.788 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.787+0000 7f9d717fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:18.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.887+0000 7f9d830b2640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:18.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.887+0000 7f9d830b2640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:18.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.887+0000 7f9d830b2640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:18.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.887+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9d7c14f3f0 con 0x7f9d7c150600 2026-02-22T21:40:18.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.890+0000 7f9d717fa640 1 -- 10.20.193.51:0/987658069 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f9d640657b0 con 0x7f9d7c150600 2026-02-22T21:40:18.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.890+0000 7f9d717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:18.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.890+0000 7f9d717fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:18.892 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:18.892 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 10 librados: watch_flush enter 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 10 librados: watch_flush exit 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 20 client.4859.objecter shutdown clearing up homeless session... 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 10 client.4859.objecter successfully canceled tick 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 msgr2=0x7f9d4c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 0x7f9d4c07b2c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c039000 tx=0x7f9d6c033000 comp rx=0 tx=0).stop 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 10 monclient: shutdown 2026-02-22T21:40:18.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 msgr2=0x7f9d7c0c3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400c4d0 tx=0x7f9d6400c9a0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 shutdown_connections 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d4c078e00 0x7f9d4c07b2c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d7c15c710 0x7f9d7c0c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d7c150600 0x7f9d7c0c3ae0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 --2- 10.20.193.51:0/987658069 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d7c150220 0x7f9d7c1f2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 >> 10.20.193.51:0/987658069 conn(0x7f9d7c0cd7b0 msgr2=0x7f9d7c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 shutdown_connections 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 -- 10.20.193.51:0/987658069 wait complete. 2026-02-22T21:40:18.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:18.891+0000 7f9d830b2640 1 librados: shutdown 2026-02-22T21:40:21.491 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:52.645 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.494 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:52.648 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.504 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:04:52.658 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.521 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:52.637 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.521 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:52.673 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.522 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:52.674 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.522 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:52.638 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.532 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:04:52.648 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.533 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:04:52.684 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:21.901 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:21.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.967+0000 7f97ba661640 10 monclient: get_monmap_and_config 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.967+0000 7f97ba661640 10 monclient: build_initial_monmap 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.967+0000 7f97ba661640 10 monclient: monmap: 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:21.968225+0000 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:21.968225+0000 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:21.970 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.967+0000 7f97ba661640 1 Processor -- start 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 1 -- start start 2026-02-22T21:40:21.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient: init 2026-02-22T21:40:21.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:21.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient: _add_conns ranks=[0,1,4,2,5,3] 2026-02-22T21:40:21.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b4150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:40:21.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f97b415c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:21.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f97b40703b0 0x7f97b40c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-b con 0x7f97b40703b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b415a920 con 0x7f97b4150220 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b415af60 con 0x7f97b415c710 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f97b415b240 con 0x7f97b40703b0 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97ba661640 10 monclient(hunting): authenticate will time out at 1297.548537s 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97b3fff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f97b4150220 0x7f97b415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:41222/0 (socket says 10.20.193.51:41222) 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.968+0000 7f97b3fff640 1 -- 10.20.193.51:0/3755665775 learned_addr learned my addr 10.20.193.51:0/3755665775 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:21.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 1 --2- 10.20.193.51:0/3755665775 >> v2:10.20.193.68:3300/0 conn(0x7f97b40703b0 0x7f97b40c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): get_auth_request con 0x7f97b40703b0 auth_method 0 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.969+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 1 -- 10.20.193.51:0/3755665775 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216212676 0 0) 0x7f97b415a920 con 0x7f97b4150220 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 10 monclient(hunting): _negotiate 2026-02-22T21:40:21.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 10 monclient(hunting): my global_id is 4908 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b2ffd640 1 -- 10.20.193.51:0/3755665775 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9798003620 con 0x7f97b4150220 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_done global_id 4888 payload 274 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 1 -- 10.20.193.51:0/3755665775 >> v1:10.20.193.68:6789/0 conn(0x7f97b415c710 legacy=0x7f97b406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 1 -- 10.20.193.51:0/3755665775 >> v1:10.20.193.51:6789/0 conn(0x7f97b4150220 legacy=0x7f97b415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 10 monclient: _finish_hunting 0 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 1 monclient: found mon.noname-b 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:21.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:21.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.970+0000 7f97b8bd7640 1 -- 10.20.193.51:0/3755665775 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b415b520 con 0x7f97b40703b0 2026-02-22T21:40:21.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.973+0000 7f97b8bd7640 1 --2- 10.20.193.51:0/3755665775 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b40703b0 0x7f97b40c73a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009dc0 tx=0x7f97a802f2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f4969d31c624092 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:21.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 1 -- 10.20.193.51:0/3755665775 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a803c070 con 0x7f97b40703b0 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: dump: 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:21.986 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 1 -- 10.20.193.51:0/3755665775 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a8037440 con 0x7f97b40703b0 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 5 monclient: authenticate success, global_id 4888 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:21.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 1 -- 10.20.193.51:0/3755665775 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8037810 con 0x7f97b40703b0 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 4 set_mon_vals no callback set 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97b2ffd640 10 monclient: dump: 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:21.988 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 10 monclient: shutdown 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 -- 10.20.193.51:0/3755665775 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b40703b0 msgr2=0x7f97b40c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 --2- 10.20.193.51:0/3755665775 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b40703b0 0x7f97b40c73a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009dc0 tx=0x7f97a802f2f0 comp rx=0 tx=0).stop 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 -- 10.20.193.51:0/3755665775 shutdown_connections 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 --2- 10.20.193.51:0/3755665775 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b40703b0 0x7f97b40c73a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 -- 10.20.193.51:0/3755665775 >> 10.20.193.51:0/3755665775 conn(0x7f97b40cd7b0 msgr2=0x7f97b40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 -- 10.20.193.51:0/3755665775 shutdown_connections 2026-02-22T21:40:21.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.975+0000 7f97ba661640 1 -- 10.20.193.51:0/3755665775 wait complete. 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 10 monclient: build_initial_monmap 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 10 monclient: monmap: 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:21.990 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 librados: starting msgr at 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 librados: starting objecter 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 Processor -- start 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 -- start start 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 librados: setting wanted keys 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 1 librados: calling monclient init 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.976+0000 7f97ba661640 10 monclient: init 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:21.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 0x7f97b41e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-a con 0x7f97b40703b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 0x7f97b414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-b con 0x7f97b4150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f97b415c710 0x7f97b41e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): picked mon.noname-c con 0x7f97b415c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4150600 con 0x7f97b40703b0 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f97b415bae0 con 0x7f97b4150220 2026-02-22T21:40:21.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4150780 con 0x7f97b415c710 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97ba661640 10 monclient(hunting): authenticate will time out at 1297.557103s 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 0x7f97b41e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 0x7f97b414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 0x7f97b41e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:58496/0 (socket says 10.20.193.51:58496) 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 1 -- 10.20.193.51:0/1426091545 learned_addr learned my addr 10.20.193.51:0/1426091545 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): get_auth_request con 0x7f97b40703b0 auth_method 0 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:21.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b8bd7640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f97b415c710 0x7f97b41e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): get_auth_request con 0x7f97b4150220 auth_method 0 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:21.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b8bd7640 10 monclient(hunting): get_auth_request con 0x7f97b415c710 auth_method 0 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b8bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b8bd7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.977+0000 7f97b8bd7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.978+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.978+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.978+0000 7f97b8bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 10 monclient(hunting): handle_auth_done global_id 4894 payload 824 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 -- 10.20.193.51:0/1426091545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f97b415c710 msgr2=0x7f97b41e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:21.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f97b415c710 0x7f97b41e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 -- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 msgr2=0x7f97b41e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 0x7f97b41e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 monclient: found mon.noname-b 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.979+0000 7f97b37fe640 1 -- 10.20.193.51:0/1426091545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41e7f80 con 0x7f97b4150220 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.980+0000 7f97b37fe640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 0x7f97b414c800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c510 tx=0x7f97a400c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.980+0000 7f97b0ff9640 10 client.?.objecter ms_handle_connect 0x7f97b4150220 2026-02-22T21:40:21.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.980+0000 7f97b0ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a40052d0 con 0x7f97b4150220 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: dump: 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:21.997 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: _finish_auth 0 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a4005470 con 0x7f97b4150220 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 5 monclient: authenticate success, global_id 4894 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4003910 con 0x7f97b4150220 2026-02-22T21:40:21.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _renew_subs 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b27fc640 4 set_mon_vals no callback set 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41e8260 con 0x7f97b4150220 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97b0ff9640 10 monclient: dump: 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:21.999 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 client.4894.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _renew_subs 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41e8820 con 0x7f97b4150220 2026-02-22T21:40:22.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 1 librados: init done 2026-02-22T21:40:22.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:22.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:22.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:22.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:21.982+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b40c73a0 con 0x7f97b4150220 2026-02-22T21:40:22.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.089+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f97a4003ab0 con 0x7f97b4150220 2026-02-22T21:40:22.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.089+0000 7f97b0ff9640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 0x7f978407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:22.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.089+0000 7f97b3fff640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 0x7f978407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:22.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.090+0000 7f97b3fff640 10 monclient: get_auth_request con 0x7f9784078e50 auth_method 0 2026-02-22T21:40:22.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.090+0000 7f97b3fff640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 0x7f978407b310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005980 tx=0x7f97a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:22.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.090+0000 7f97b0ff9640 10 client.4894.objecter ms_handle_connect 0x7f9784078e50 2026-02-22T21:40:22.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.091+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f97a405df90 con 0x7f97b4150220 2026-02-22T21:40:22.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.091+0000 7f97b0ff9640 10 client.4894.objecter ms_dispatch2 0x7f97b4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:22.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.091+0000 7f97b0ff9640 3 client.4894.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:22.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.091+0000 7f97b0ff9640 3 client.4894.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:22.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.091+0000 7f97b0ff9640 20 client.4894.objecter dump_active .. 0 homeless 2026-02-22T21:40:22.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.098+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f97a4061f50 con 0x7f97b4150220 2026-02-22T21:40:22.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.098+0000 7f97b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:22.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.098+0000 7f97b0ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:22.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.197+0000 7f97ba661640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:22.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.197+0000 7f97ba661640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:22.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.197+0000 7f97ba661640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:22.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.197+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f97b4074da0 con 0x7f97b4150220 2026-02-22T21:40:22.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.200+0000 7f97b0ff9640 1 -- 10.20.193.51:0/1426091545 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f97a4066250 con 0x7f97b4150220 2026-02-22T21:40:22.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.200+0000 7f97b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:22.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.200+0000 7f97b0ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:22.201 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:22.201 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 10 librados: watch_flush enter 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 10 librados: watch_flush exit 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 20 client.4894.objecter shutdown clearing up homeless session... 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 10 client.4894.objecter successfully canceled tick 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 msgr2=0x7f978407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 0x7f978407b310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005980 tx=0x7f97a0009040 comp rx=0 tx=0).stop 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 10 monclient: shutdown 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 msgr2=0x7f97b414c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:22.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 0x7f97b414c800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c510 tx=0x7f97a400c9e0 comp rx=0 tx=0).stop 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 shutdown_connections 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9784078e50 0x7f978407b310 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f97b415c710 0x7f97b41e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f97b4150220 0x7f97b414c800 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 --2- 10.20.193.51:0/1426091545 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f97b40703b0 0x7f97b41e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 >> 10.20.193.51:0/1426091545 conn(0x7f97b40cd7b0 msgr2=0x7f97b40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 shutdown_connections 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 -- 10.20.193.51:0/1426091545 wait complete. 2026-02-22T21:40:22.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:22.201+0000 7f97ba661640 1 librados: shutdown 2026-02-22T21:40:25.212 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:25.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.279+0000 7f682f796640 10 monclient: get_monmap_and_config 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.279+0000 7f682f796640 10 monclient: build_initial_monmap 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.279+0000 7f682f796640 10 monclient: monmap: 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:25.280573+0000 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:25.280573+0000 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:25.282 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 Processor -- start 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 -- start start 2026-02-22T21:40:25.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient: init 2026-02-22T21:40:25.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient: _add_conns ranks=[1,4,2,3,0,5] 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f6828150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f682815c710 0x7f68280c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): picked mon.noname-b con 0x7f682815c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6828150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682815a920 con 0x7f6828150220 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682815af60 con 0x7f6828150600 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f682815b240 con 0x7f682815c710 2026-02-22T21:40:25.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.280+0000 7f682f796640 10 monclient(hunting): authenticate will time out at 1300.860901s 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682cd0a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f682815c710 0x7f68280c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682dd0c640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f6828150600 0x7f682806fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:48570/0 (socket says 10.20.193.51:48570) 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682dd0c640 1 -- 10.20.193.51:0/1612106719 learned_addr learned my addr 10.20.193.51:0/1612106719 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682cd0a640 10 monclient(hunting): get_auth_request con 0x7f682815c710 auth_method 0 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682cd0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682cd0a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.281+0000 7f682cd0a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:25.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 1 -- 10.20.193.51:0/1612106719 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744423432 0 0) 0x7f682815af60 con 0x7f6828150600 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 10 monclient(hunting): _negotiate 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 10 monclient(hunting): my global_id is 4865 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.282+0000 7f6817fff640 1 -- 10.20.193.51:0/1612106719 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6810003620 con 0x7f6828150600 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 10 monclient(hunting): handle_auth_done global_id 4897 payload 274 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 1 -- 10.20.193.51:0/1612106719 >> v1:10.20.193.125:6789/0 conn(0x7f6828150600 legacy=0x7f682806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 1 -- 10.20.193.51:0/1612106719 >> v1:10.20.193.68:6789/0 conn(0x7f6828150220 legacy=0x7f682815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 10 monclient: _finish_hunting 0 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 1 monclient: found mon.noname-b 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:25.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.283+0000 7f682cd0a640 1 -- 10.20.193.51:0/1612106719 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682815b520 con 0x7f682815c710 2026-02-22T21:40:25.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.285+0000 7f682cd0a640 1 --2- 10.20.193.51:0/1612106719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f682815c710 0x7f68280c73a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098d0 tx=0x7f681c02ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a47a57282613045 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:25.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 1 -- 10.20.193.51:0/1612106719 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c03c070 con 0x7f682815c710 2026-02-22T21:40:25.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:25.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: dump: 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:25.297 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 1 -- 10.20.193.51:0/1612106719 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681c02fbf0 con 0x7f682815c710 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 1 -- 10.20.193.51:0/1612106719 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c037440 con 0x7f682815c710 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f682f796640 5 monclient: authenticate success, global_id 4897 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f682f796640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f682f796640 4 set_mon_vals no callback set 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:25.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.286+0000 7f6817fff640 10 monclient: dump: 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 10 monclient: shutdown 2026-02-22T21:40:25.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 -- 10.20.193.51:0/1612106719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f682815c710 msgr2=0x7f68280c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 --2- 10.20.193.51:0/1612106719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f682815c710 0x7f68280c73a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098d0 tx=0x7f681c02ef20 comp rx=0 tx=0).stop 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 -- 10.20.193.51:0/1612106719 shutdown_connections 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 --2- 10.20.193.51:0/1612106719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f682815c710 0x7f68280c73a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 -- 10.20.193.51:0/1612106719 >> 10.20.193.51:0/1612106719 conn(0x7f68280cd7b0 msgr2=0x7f68280cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 -- 10.20.193.51:0/1612106719 shutdown_connections 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 1 -- 10.20.193.51:0/1612106719 wait complete. 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 10 monclient: build_initial_monmap 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.287+0000 7f682f796640 10 monclient: monmap: 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:25.300 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 librados: starting msgr at 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 librados: starting objecter 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 Processor -- start 2026-02-22T21:40:25.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 -- start start 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 librados: setting wanted keys 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 librados: calling monclient init 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient: init 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): picked mon.noname-b con 0x7f6828150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6828150600 0x7f68280c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): picked mon.noname-c con 0x7f6828150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f682815c710 0x7f68281f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:25.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): picked mon.noname-a con 0x7f682815c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f68280703b0 con 0x7f682815c710 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f682815bae0 con 0x7f6828150220 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6828070530 con 0x7f6828150600 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682f796640 10 monclient(hunting): authenticate will time out at 1300.868617s 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.288+0000 7f682d50b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f682815c710 0x7f68281f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6828150600 0x7f68280c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:25.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47808/0 (socket says 10.20.193.51:47808) 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 1 -- 10.20.193.51:0/620925490 learned_addr learned my addr 10.20.193.51:0/620925490 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): get_auth_request con 0x7f6828150220 auth_method 0 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): get_auth_request con 0x7f682815c710 auth_method 0 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): get_auth_request con 0x7f6828150600 auth_method 0 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:25.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682d50b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682cd0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.289+0000 7f682dd0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 -- 10.20.193.51:0/620925490 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6828150600 msgr2=0x7f68280c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6828150600 0x7f68280c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 -- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f682815c710 msgr2=0x7f68281f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f682815c710 0x7f68281f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 10 monclient: _finish_hunting 0 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 monclient: found mon.noname-b 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.290+0000 7f682d50b640 1 -- 10.20.193.51:0/620925490 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68280c19d0 con 0x7f6828150220 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.291+0000 7f682d50b640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f681800c700 tx=0x7f681800cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.291+0000 7f6815ffb640 10 client.?.objecter ms_handle_connect 0x7f6828150220 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.291+0000 7f6815ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:25.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68180052d0 con 0x7f6828150220 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: dump: 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:25.307 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: _finish_auth 0 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 5 monclient: authenticate success, global_id 4903 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _renew_subs 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68280c1cb0 con 0x7f6828150220 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 client.4903.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:25.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _renew_subs 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68281f3280 con 0x7f6828150220 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 1 librados: init done 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818005470 con 0x7f6828150220 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f68177fe640 4 set_mon_vals no callback set 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681800a840 con 0x7f6828150220 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f6815ffb640 10 monclient: dump: 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:25.309 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.310 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.293+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6828074da0 con 0x7f6828150220 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.304+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f681800a9e0 con 0x7f6828150220 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.304+0000 7f6815ffb640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 0x7f67f807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.305+0000 7f682cd0a640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 0x7f67f807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.305+0000 7f682cd0a640 10 monclient: get_auth_request con 0x7f67f8078e00 auth_method 0 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.305+0000 7f682cd0a640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 0x7f67f807b2c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f681c039000 tx=0x7f681c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.305+0000 7f6815ffb640 10 client.4903.objecter ms_handle_connect 0x7f67f8078e00 2026-02-22T21:40:25.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.306+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f681800ad00 con 0x7f6828150220 2026-02-22T21:40:25.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.306+0000 7f6815ffb640 10 client.4903.objecter ms_dispatch2 0x7f6828000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:25.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.306+0000 7f6815ffb640 3 client.4903.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:25.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.306+0000 7f6815ffb640 3 client.4903.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:25.313 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.306+0000 7f6815ffb640 20 client.4903.objecter dump_active .. 0 homeless 2026-02-22T21:40:25.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.405+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6818062000 con 0x7f6828150220 2026-02-22T21:40:25.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.405+0000 7f6815ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:25.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.405+0000 7f6815ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:25.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.505+0000 7f682f796640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:25.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.505+0000 7f682f796640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:25.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.505+0000 7f682f796640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:40:25.507 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.505+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f682814ef60 con 0x7f6828150220 2026-02-22T21:40:25.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.508+0000 7f6815ffb640 1 -- 10.20.193.51:0/620925490 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f6818066300 con 0x7f6828150220 2026-02-22T21:40:25.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.508+0000 7f6815ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:25.509 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.508+0000 7f6815ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:25.509 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:25.509 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:25.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 10 librados: watch_flush enter 2026-02-22T21:40:25.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 10 librados: watch_flush exit 2026-02-22T21:40:25.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 20 client.4903.objecter shutdown clearing up homeless session... 2026-02-22T21:40:25.510 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 10 client.4903.objecter successfully canceled tick 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 msgr2=0x7f67f807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 0x7f67f807b2c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f681c039000 tx=0x7f681c033000 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 10 monclient: shutdown 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 msgr2=0x7f68280c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f681800c700 tx=0x7f681800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 shutdown_connections 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f67f8078e00 0x7f67f807b2c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f682815c710 0x7f68281f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6828150600 0x7f68280c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 --2- 10.20.193.51:0/620925490 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6828150220 0x7f68280c10d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 >> 10.20.193.51:0/620925490 conn(0x7f68280cd7b0 msgr2=0x7f68280c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 shutdown_connections 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 -- 10.20.193.51:0/620925490 wait complete. 2026-02-22T21:40:25.511 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:25.509+0000 7f682f796640 1 librados: shutdown 2026-02-22T21:40:28.520 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 10 monclient: get_monmap_and_config 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 10 monclient: build_initial_monmap 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 10 monclient: monmap: 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:28.587207+0000 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:28.587207+0000 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 1 Processor -- start 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 1 -- start start 2026-02-22T21:40:28.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.586+0000 7fcfce0bd640 10 monclient: init 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient: _add_conns ranks=[4,3,2,5,0,1] 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fcfc8150220 0x7fcfc80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-b con 0x7fcfc8150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcfc8150600 0x7fcfc815c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-a con 0x7fcfc8150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfc815c540 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfc815c9b0 con 0x7fcfc815c540 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fcfc815a920 con 0x7fcfc8150600 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fcfc815af60 con 0x7fcfc8150220 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfce0bd640 10 monclient(hunting): authenticate will time out at 1304.167496s 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfc6ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcfc8150600 0x7fcfc815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfc77fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fcfc8150220 0x7fcfc80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.587+0000 7fcfc7fff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fcfc815c540 0x7fcfc806fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:42758/0 (socket says 10.20.193.51:42758) 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc7fff640 1 -- 10.20.193.51:0/2936687897 learned_addr learned my addr 10.20.193.51:0/2936687897 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): get_auth_request con 0x7fcfc8150600 auth_method 0 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:28.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): get_auth_request con 0x7fcfc8150220 auth_method 0 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.588+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_done global_id 4917 payload 274 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 1 -- 10.20.193.51:0/2936687897 >> v2:10.20.193.68:3300/0 conn(0x7fcfc8150220 msgr2=0x7fcfc80c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 1 --2- 10.20.193.51:0/2936687897 >> v2:10.20.193.68:3300/0 conn(0x7fcfc8150220 0x7fcfc80c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 1 -- 10.20.193.51:0/2936687897 >> v1:10.20.193.125:6789/0 conn(0x7fcfc815c540 legacy=0x7fcfc806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 1 monclient: found mon.noname-a 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.589+0000 7fcfc6ffd640 1 -- 10.20.193.51:0/2936687897 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc815b240 con 0x7fcfc8150600 2026-02-22T21:40:28.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.590+0000 7fcfc6ffd640 1 --2- 10.20.193.51:0/2936687897 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc8150600 0x7fcfc815c000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8070940 tx=0x7fcfb802eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a22d231ffa20d91 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 1 -- 10.20.193.51:0/2936687897 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb803c070 con 0x7fcfc8150600 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: dump: 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:28.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 1 -- 10.20.193.51:0/2936687897 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb802fb80 con 0x7fcfc8150600 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 5 monclient: authenticate success, global_id 4917 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 1 -- 10.20.193.51:0/2936687897 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8037440 con 0x7fcfc8150600 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 4 set_mon_vals no callback set 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfc67fc640 10 monclient: dump: 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:28.594 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 10 monclient: shutdown 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 -- 10.20.193.51:0/2936687897 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc8150600 msgr2=0x7fcfc815c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/2936687897 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc8150600 0x7fcfc815c000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8070940 tx=0x7fcfb802eeb0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 -- 10.20.193.51:0/2936687897 shutdown_connections 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/2936687897 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc8150600 0x7fcfc815c000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/2936687897 >> v2:10.20.193.68:3300/0 conn(0x7fcfc8150220 0x7fcfc80c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 -- 10.20.193.51:0/2936687897 >> 10.20.193.51:0/2936687897 conn(0x7fcfc80cd7b0 msgr2=0x7fcfc80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 -- 10.20.193.51:0/2936687897 shutdown_connections 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.592+0000 7fcfce0bd640 1 -- 10.20.193.51:0/2936687897 wait complete. 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient: build_initial_monmap 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient: monmap: 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 librados: starting msgr at 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 librados: starting objecter 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 Processor -- start 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 -- start start 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 librados: setting wanted keys 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 librados: calling monclient init 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient: init 2026-02-22T21:40:28.595 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfc8150220 0x7fcfc80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-b con 0x7fcfc8150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfc8150600 0x7fcfc81f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-c con 0x7fcfc8150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): picked mon.noname-a con 0x7fcfc815c540 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc815bae0 con 0x7fcfc815c540 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc815b800 con 0x7fcfc8150220 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc80703b0 con 0x7fcfc8150600 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:28.596 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.593+0000 7fcfce0bd640 10 monclient(hunting): authenticate will time out at 1304.173936s 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:56954/0 (socket says 10.20.193.51:56954) 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 1 -- 10.20.193.51:0/43948354 learned_addr learned my addr 10.20.193.51:0/43948354 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): get_auth_request con 0x7fcfc815c540 auth_method 0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfc8150600 0x7fcfc81f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfc8150220 0x7fcfc80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): get_auth_request con 0x7fcfc8150220 auth_method 0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): get_auth_request con 0x7fcfc8150600 auth_method 0 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:28.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.594+0000 7fcfc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 10 monclient(hunting): handle_auth_done global_id 4920 payload 824 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 -- 10.20.193.51:0/43948354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfc8150600 msgr2=0x7fcfc81f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfc8150600 0x7fcfc81f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 -- 10.20.193.51:0/43948354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfc8150220 msgr2=0x7fcfc80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfc8150220 0x7fcfc80c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 monclient: found mon.noname-a 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.595+0000 7fcfc7fff640 1 -- 10.20.193.51:0/43948354 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc80c9ff0 con 0x7fcfc815c540 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.596+0000 7fcfc7fff640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00cb10 tx=0x7fcfbc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.596+0000 7fcfa3fff640 10 client.?.objecter ms_handle_connect 0x7fcfc815c540 2026-02-22T21:40:28.598 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.596+0000 7fcfa3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc017070 con 0x7fcfc815c540 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: dump: 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:28.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfbc00ac50 con 0x7fcfc815c540 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 5 monclient: authenticate success, global_id 4920 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _renew_subs 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc80ca2d0 con 0x7fcfc815c540 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc012600 con 0x7fcfc815c540 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfc5ffb640 4 set_mon_vals no callback set 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfa3fff640 10 monclient: dump: 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 client.4920.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _renew_subs 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc80ca890 con 0x7fcfc815c540 2026-02-22T21:40:28.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 1 librados: init done 2026-02-22T21:40:28.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:28.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:28.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.598+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc815c000 con 0x7fcfc815c540 2026-02-22T21:40:28.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.610+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcfbc0127a0 con 0x7fcfc815c540 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.610+0000 7fcfa3fff640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 0x7fcf9807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.610+0000 7fcfc77fe640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 0x7fcf9807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.610+0000 7fcfc77fe640 10 monclient: get_auth_request con 0x7fcf98078e50 auth_method 0 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 10 client.4920.objecter ms_handle_connect 0x7fcf98078e50 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfc77fe640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 0x7fcf9807b310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009c40 tx=0x7fcfb4009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:28.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fcfbc05dc40 con 0x7fcfc815c540 2026-02-22T21:40:28.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 10 client.4920.objecter ms_dispatch2 0x7fcfc8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:28.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 3 client.4920.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:28.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 3 client.4920.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:28.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.611+0000 7fcfa3fff640 20 client.4920.objecter dump_active .. 0 homeless 2026-02-22T21:40:28.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.711+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fcfbc022020 con 0x7fcfc815c540 2026-02-22T21:40:28.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.711+0000 7fcfa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:28.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.711+0000 7fcfa3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:28.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.813+0000 7fcfce0bd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:28.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.813+0000 7fcfce0bd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:28.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.813+0000 7fcfce0bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:28.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.813+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcfc8074da0 con 0x7fcfc815c540 2026-02-22T21:40:28.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.816+0000 7fcfa3fff640 1 -- 10.20.193.51:0/43948354 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fcfbc061c00 con 0x7fcfc815c540 2026-02-22T21:40:28.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.816+0000 7fcfa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:28.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.816+0000 7fcfa3fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:28.817 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:28.817 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 10 librados: watch_flush enter 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 10 librados: watch_flush exit 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 20 client.4920.objecter shutdown clearing up homeless session... 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 10 client.4920.objecter successfully canceled tick 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 msgr2=0x7fcf9807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 0x7fcf9807b310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009c40 tx=0x7fcfb4009290 comp rx=0 tx=0).stop 2026-02-22T21:40:28.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 10 monclient: shutdown 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 msgr2=0x7fcfc81e7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00cb10 tx=0x7fcfbc00a060 comp rx=0 tx=0).stop 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 shutdown_connections 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcf98078e50 0x7fcf9807b310 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcfc815c540 0x7fcfc81e7340 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcfc8150600 0x7fcfc81f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 --2- 10.20.193.51:0/43948354 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcfc8150220 0x7fcfc80c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 >> 10.20.193.51:0/43948354 conn(0x7fcfc80cd7b0 msgr2=0x7fcfc806eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 shutdown_connections 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 -- 10.20.193.51:0/43948354 wait complete. 2026-02-22T21:40:28.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:28.817+0000 7fcfce0bd640 1 librados: shutdown 2026-02-22T21:40:31.828 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:31.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.892+0000 7fdfc7bed640 10 monclient: get_monmap_and_config 2026-02-22T21:40:31.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.892+0000 7fdfc7bed640 10 monclient: build_initial_monmap 2026-02-22T21:40:31.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.892+0000 7fdfc7bed640 10 monclient: monmap: 2026-02-22T21:40:31.894 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:31.893318+0000 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:31.893318+0000 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:31.895 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:31.896 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 Processor -- start 2026-02-22T21:40:31.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 -- start start 2026-02-22T21:40:31.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient: init 2026-02-22T21:40:31.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:31.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient: _add_conns ranks=[5,1,4,0,2,3] 2026-02-22T21:40:31.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdfc01527b0 0x7fdfc00c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc01527b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fdfc00bf800 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fdfc00703b0 0x7fdfc015f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc00703b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc015ffa0 con 0x7fdfc00bf800 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc00be810 con 0x7fdfc00703b0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc0152b90 con 0x7fdfc01527b0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.893+0000 7fdfc7bed640 10 monclient(hunting): authenticate will time out at 1307.473661s 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdfc01527b0 0x7fdfc00c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fdfc00703b0 0x7fdfc015f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdfc01527b0 0x7fdfc00c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:46904/0 (socket says 10.20.193.51:46904) 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 1 -- 10.20.193.51:0/1854299706 learned_addr learned my addr 10.20.193.51:0/1854299706 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): get_auth_request con 0x7fdfc01527b0 auth_method 0 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): get_auth_request con 0x7fdfc00703b0 auth_method 0 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:31.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.894+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 10 monclient(hunting): handle_auth_done global_id 4877 payload 274 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 1 -- 10.20.193.51:0/1854299706 >> v2:10.20.193.68:3300/0 conn(0x7fdfc00703b0 msgr2=0x7fdfc015f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 1 --2- 10.20.193.51:0/1854299706 >> v2:10.20.193.68:3300/0 conn(0x7fdfc00703b0 0x7fdfc015f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 1 -- 10.20.193.51:0/1854299706 >> v1:10.20.193.68:6789/0 conn(0x7fdfc00bf800 legacy=0x7fdfc006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 10 monclient: _finish_hunting 0 2026-02-22T21:40:31.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 1 monclient: found mon.noname-c 2026-02-22T21:40:31.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:31.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:31.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.895+0000 7fdfc5962640 1 -- 10.20.193.51:0/1854299706 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc00bea60 con 0x7fdfc01527b0 2026-02-22T21:40:31.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.896+0000 7fdfc5962640 1 --2- 10.20.193.51:0/1854299706 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c9660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00c8f90 tx=0x7fdfb002eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb32906edc9c859 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:31.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 1 -- 10.20.193.51:0/1854299706 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb003c070 con 0x7fdfc01527b0 2026-02-22T21:40:31.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: dump: 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:31.909 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: _finish_auth 0 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: _check_auth_tickets 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 1 -- 10.20.193.51:0/1854299706 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb002fbe0 con 0x7fdfc01527b0 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 1 -- 10.20.193.51:0/1854299706 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0037440 con 0x7fdfc01527b0 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc4960640 10 monclient: dump: 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:31.910 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 5 monclient: authenticate success, global_id 4877 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 4 set_mon_vals no callback set 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 10 monclient: shutdown 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:31.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 1 -- 10.20.193.51:0/1854299706 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 msgr2=0x7fdfc00c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/1854299706 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c9660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00c8f90 tx=0x7fdfb002eed0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 1 -- 10.20.193.51:0/1854299706 shutdown_connections 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/1854299706 >> v2:10.20.193.68:3300/0 conn(0x7fdfc00703b0 0x7fdfc015f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.898+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/1854299706 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c9660 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 -- 10.20.193.51:0/1854299706 >> 10.20.193.51:0/1854299706 conn(0x7fdfc00cd7b0 msgr2=0x7fdfc00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 -- 10.20.193.51:0/1854299706 shutdown_connections 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 -- 10.20.193.51:0/1854299706 wait complete. 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 10 monclient: build_initial_monmap 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 10 monclient: monmap: 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 librados: starting msgr at 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 librados: starting objecter 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 Processor -- start 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 -- start start 2026-02-22T21:40:31.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 librados: setting wanted keys 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 1 librados: calling monclient init 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.899+0000 7fdfc7bed640 10 monclient: init 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdfc00703b0 0x7fdfc00c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc00703b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdfc00bf800 0x7fdfc01f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc00bf800 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc01527b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc00bf300 con 0x7fdfc00703b0 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc00bf020 con 0x7fdfc00bf800 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc006d710 con 0x7fdfc01527b0 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc7bed640 10 monclient(hunting): authenticate will time out at 1307.480343s 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc5161640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdfc00bf800 0x7fdfc01f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc5962640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdfc00703b0 0x7fdfc00c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:46912/0 (socket says 10.20.193.51:46912) 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 1 -- 10.20.193.51:0/3192506852 learned_addr learned my addr 10.20.193.51:0/3192506852 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:31.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 10 monclient(hunting): get_auth_request con 0x7fdfc01527b0 auth_method 0 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.900+0000 7fdfc6163640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): get_auth_request con 0x7fdfc00bf800 auth_method 0 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc6163640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): get_auth_request con 0x7fdfc00703b0 auth_method 0 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5161640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.901+0000 7fdfc5962640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 10 monclient(hunting): handle_auth_done global_id 4880 payload 824 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 -- 10.20.193.51:0/3192506852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdfc00bf800 msgr2=0x7fdfc01f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdfc00bf800 0x7fdfc01f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 -- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdfc00703b0 msgr2=0x7fdfc00c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:31.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdfc00703b0 0x7fdfc00c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 10 monclient: _finish_hunting 0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 monclient: found mon.noname-c 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.902+0000 7fdfc6163640 1 -- 10.20.193.51:0/3192506852 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc015e080 con 0x7fdfc01527b0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.903+0000 7fdfc6163640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00efc0 tx=0x7fdfbc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.903+0000 7fdfae7fc640 10 client.?.objecter ms_handle_connect 0x7fdfc01527b0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.903+0000 7fdfae7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc018020 con 0x7fdfc01527b0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 10 monclient: dump: 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:31.915 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 5 monclient: authenticate success, global_id 4880 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 10 monclient: _renew_subs 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc015e360 con 0x7fdfc01527b0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 10 client.4880.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 10 monclient: _renew_subs 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc015e920 con 0x7fdfc01527b0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.904+0000 7fdfc7bed640 1 librados: init done 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfbc014480 con 0x7fdfc01527b0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc014830 con 0x7fdfc01527b0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfae7fc640 10 monclient: dump: 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:31.916 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfaffff640 4 set_mon_vals no callback set 2026-02-22T21:40:31.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfc7bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:31.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfc7bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:31.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfc7bed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:31.917 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.905+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc00bfc90 con 0x7fdfc01527b0 2026-02-22T21:40:31.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fdfbc0149d0 con 0x7fdfc01527b0 2026-02-22T21:40:31.919 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfae7fc640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 0x7fdf9007b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:31.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfc5962640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 0x7fdf9007b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:31.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfc5962640 10 monclient: get_auth_request con 0x7fdf90078e30 auth_method 0 2026-02-22T21:40:31.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfae7fc640 10 client.4880.objecter ms_handle_connect 0x7fdf90078e30 2026-02-22T21:40:31.920 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.918+0000 7fdfc5962640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 0x7fdf9007b2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00709e0 tx=0x7fdfb0039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:31.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.920+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fdfbc05ee10 con 0x7fdfc01527b0 2026-02-22T21:40:31.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.920+0000 7fdfae7fc640 10 client.4880.objecter ms_dispatch2 0x7fdfc0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:31.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.920+0000 7fdfae7fc640 3 client.4880.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:31.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.920+0000 7fdfae7fc640 3 client.4880.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:31.921 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:31.920+0000 7fdfae7fc640 20 client.4880.objecter dump_active .. 0 homeless 2026-02-22T21:40:32.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.023+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fdfbc022030 con 0x7fdfc01527b0 2026-02-22T21:40:32.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.023+0000 7fdfae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:32.025 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.023+0000 7fdfae7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:32.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.124+0000 7fdfc7bed640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:32.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.124+0000 7fdfc7bed640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:32.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.124+0000 7fdfc7bed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:32.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.124+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdfc0074e60 con 0x7fdfc01527b0 2026-02-22T21:40:32.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfae7fc640 1 -- 10.20.193.51:0/3192506852 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fdfbc010040 con 0x7fdfc01527b0 2026-02-22T21:40:32.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:32.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfae7fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:32.128 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:32.128 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 10 librados: watch_flush enter 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 10 librados: watch_flush exit 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 20 client.4880.objecter shutdown clearing up homeless session... 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 10 client.4880.objecter successfully canceled tick 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 msgr2=0x7fdf9007b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 0x7fdf9007b2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00709e0 tx=0x7fdfb0039040 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 10 monclient: shutdown 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 msgr2=0x7fdfc00c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.127+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00efc0 tx=0x7fdfbc00c490 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 shutdown_connections 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdf90078e30 0x7fdf9007b2f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdfc01527b0 0x7fdfc00c6cf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdfc00bf800 0x7fdfc01f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 --2- 10.20.193.51:0/3192506852 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdfc00703b0 0x7fdfc00c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 >> 10.20.193.51:0/3192506852 conn(0x7fdfc00cd7b0 msgr2=0x7fdfc00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 shutdown_connections 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 -- 10.20.193.51:0/3192506852 wait complete. 2026-02-22T21:40:32.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:32.128+0000 7fdfc7bed640 1 librados: shutdown 2026-02-22T21:40:35.137 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 10 monclient: get_monmap_and_config 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 10 monclient: build_initial_monmap 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 10 monclient: monmap: 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:35.204813+0000 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:35.204813+0000 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:35.205 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 1 Processor -- start 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 1 -- start start 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.203+0000 7f81d6205640 10 monclient: init 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient: _add_conns ranks=[4,3,2,1,0,5] 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f81d01527b0 0x7f81d00c9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-b con 0x7f81d01527b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f81d00bf800 0x7f81d006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-a con 0x7f81d00bf800 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81d00bfc90 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d015ffa0 con 0x7f81d00bfc90 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f81d00be810 con 0x7f81d00bf800 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:35.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f81d0152b90 con 0x7f81d01527b0 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81d6205640 10 monclient(hunting): authenticate will time out at 1310.785151s 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cf7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f81d01527b0 0x7f81d00c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81ceffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f81d00bf800 0x7f81d006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cffff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f81d00bfc90 0x7f81d015f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:42770/0 (socket says 10.20.193.51:42770) 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cffff640 1 -- 10.20.193.51:0/2948210369 learned_addr learned my addr 10.20.193.51:0/2948210369 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81ceffd640 10 monclient(hunting): get_auth_request con 0x7f81d00bf800 auth_method 0 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81ceffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81ceffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d01527b0 auth_method 0 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.204+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:35.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.205+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 1 -- 10.20.193.51:0/2948210369 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771198059 0 0) 0x7f81d015ffa0 con 0x7f81d00bfc90 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 10 monclient(hunting): my global_id is 4883 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ce7fc640 1 -- 10.20.193.51:0/2948210369 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81b4003620 con 0x7f81d00bfc90 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 10 monclient(hunting): handle_auth_done global_id 4926 payload 274 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 1 -- 10.20.193.51:0/2948210369 >> v2:10.20.193.68:3300/0 conn(0x7f81d01527b0 msgr2=0x7f81d00c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 1 --2- 10.20.193.51:0/2948210369 >> v2:10.20.193.68:3300/0 conn(0x7f81d01527b0 0x7f81d00c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 1 -- 10.20.193.51:0/2948210369 >> v1:10.20.193.125:6789/0 conn(0x7f81d00bfc90 legacy=0x7f81d015f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 10 monclient: _finish_hunting 0 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 1 monclient: found mon.noname-a 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.206+0000 7f81ceffd640 1 -- 10.20.193.51:0/2948210369 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d00bea60 con 0x7f81d00bf800 2026-02-22T21:40:35.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.207+0000 7f81ceffd640 1 --2- 10.20.193.51:0/2948210369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d00bf800 0x7f81d006fca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f81d00c8f90 tx=0x7f81c002eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181ba6551e1d364 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 1 -- 10.20.193.51:0/2948210369 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c003c070 con 0x7f81d00bf800 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: dump: 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 1 -- 10.20.193.51:0/2948210369 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c002fb80 con 0x7f81d00bf800 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 1 -- 10.20.193.51:0/2948210369 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0037440 con 0x7f81d00bf800 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81ce7fc640 10 monclient: dump: 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:35.211 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 5 monclient: authenticate success, global_id 4926 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 4 set_mon_vals no callback set 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 10 monclient: shutdown 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 -- 10.20.193.51:0/2948210369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d00bf800 msgr2=0x7f81d006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 --2- 10.20.193.51:0/2948210369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d00bf800 0x7f81d006fca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f81d00c8f90 tx=0x7f81c002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 -- 10.20.193.51:0/2948210369 shutdown_connections 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 --2- 10.20.193.51:0/2948210369 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d00bf800 0x7f81d006fca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 --2- 10.20.193.51:0/2948210369 >> v2:10.20.193.68:3300/0 conn(0x7f81d01527b0 0x7f81d00c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.212 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 -- 10.20.193.51:0/2948210369 >> 10.20.193.51:0/2948210369 conn(0x7f81d00cd7b0 msgr2=0x7f81d00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 -- 10.20.193.51:0/2948210369 shutdown_connections 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.209+0000 7f81d6205640 1 -- 10.20.193.51:0/2948210369 wait complete. 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient: build_initial_monmap 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient: monmap: 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 librados: starting msgr at 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 librados: starting objecter 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 Processor -- start 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 -- start start 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 librados: setting wanted keys 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 librados: calling monclient init 2026-02-22T21:40:35.213 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient: init 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81d00bf800 0x7f81d01f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-c con 0x7f81d00bf800 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81d00bfc90 0x7f81d01f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-b con 0x7f81d00bfc90 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): picked mon.noname-a con 0x7f81d01527b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00bf300 con 0x7f81d01527b0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00bf020 con 0x7f81d00bfc90 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f81d006d710 con 0x7f81d00bf800 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81d6205640 10 monclient(hunting): authenticate will time out at 1310.791764s 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.210+0000 7f81cffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:57002/0 (socket says 10.20.193.51:57002) 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 1 -- 10.20.193.51:0/1454485624 learned_addr learned my addr 10.20.193.51:0/1454485624 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d01527b0 auth_method 0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81d00bfc90 0x7f81d01f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81d00bf800 0x7f81d01f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): get_auth_request con 0x7f81d00bfc90 auth_method 0 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request con 0x7f81d00bf800 auth_method 0 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.211+0000 7f81cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 10 monclient(hunting): handle_auth_done global_id 4929 payload 824 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 -- 10.20.193.51:0/1454485624 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81d00bf800 msgr2=0x7f81d01f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81d00bf800 0x7f81d01f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 -- 10.20.193.51:0/1454485624 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81d00bfc90 msgr2=0x7f81d01f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81d00bfc90 0x7f81d01f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 monclient: found mon.noname-a 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.212+0000 7f81cffff640 1 -- 10.20.193.51:0/1454485624 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d015f230 con 0x7f81d01527b0 2026-02-22T21:40:35.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.213+0000 7f81cffff640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f81c400caa0 tx=0x7f81c400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:35.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.213+0000 7f81abfff640 10 client.?.objecter ms_handle_connect 0x7f81d01527b0 2026-02-22T21:40:35.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.213+0000 7f81abfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:35.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c400ac40 con 0x7f81d01527b0 2026-02-22T21:40:35.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:35.217 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: dump: 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: _finish_auth 0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c400ade0 con 0x7f81d01527b0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4012600 con 0x7f81d01527b0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81cdffb640 4 set_mon_vals no callback set 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81abfff640 10 monclient: dump: 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 5 monclient: authenticate success, global_id 4929 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 10 monclient: _renew_subs 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d01f3d70 con 0x7f81d01527b0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 10 client.4929.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 10 monclient: _renew_subs 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d01f4330 con 0x7f81d01527b0 2026-02-22T21:40:35.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.215+0000 7f81d6205640 1 librados: init done 2026-02-22T21:40:35.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.216+0000 7f81d6205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:35.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.216+0000 7f81d6205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:35.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.216+0000 7f81d6205640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.216+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d00c9660 con 0x7f81d01527b0 2026-02-22T21:40:35.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.227+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f81c40127a0 con 0x7f81d01527b0 2026-02-22T21:40:35.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.227+0000 7f81abfff640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 0x7f81a007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:35.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.227+0000 7f81cf7fe640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 0x7f81a007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:35.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.227+0000 7f81cf7fe640 10 monclient: get_auth_request con 0x7f81a0078e50 auth_method 0 2026-02-22T21:40:35.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.228+0000 7f81abfff640 10 client.4929.objecter ms_handle_connect 0x7f81a0078e50 2026-02-22T21:40:35.230 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.228+0000 7f81cf7fe640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 0x7f81a007b310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f81d014c2d0 tx=0x7f81bc009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f81c405e800 con 0x7f81d01527b0 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 10 client.4929.objecter ms_dispatch2 0x7f81d0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 3 client.4929.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 3 client.4929.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 20 client.4929.objecter dump_active .. 0 homeless 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f81c40627c0 con 0x7f81d01527b0 2026-02-22T21:40:35.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:35.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.331+0000 7f81abfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:35.434 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.432+0000 7f81d6205640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:35.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.432+0000 7f81d6205640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:35.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.432+0000 7f81d6205640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:35.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.432+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f81d006e120 con 0x7f81d01527b0 2026-02-22T21:40:35.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.434+0000 7f81abfff640 1 -- 10.20.193.51:0/1454485624 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f81c4066ac0 con 0x7f81d01527b0 2026-02-22T21:40:35.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.434+0000 7f81abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:35.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.434+0000 7f81abfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:35.437 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:35.437 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 10 librados: watch_flush enter 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 10 librados: watch_flush exit 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 20 client.4929.objecter shutdown clearing up homeless session... 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 10 client.4929.objecter successfully canceled tick 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 msgr2=0x7f81a007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 0x7f81a007b310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f81d014c2d0 tx=0x7f81bc009290 comp rx=0 tx=0).stop 2026-02-22T21:40:35.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 10 monclient: shutdown 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 msgr2=0x7f81d015ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f81c400caa0 tx=0x7f81c400cf70 comp rx=0 tx=0).stop 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 shutdown_connections 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f81a0078e50 0x7f81a007b310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f81d01527b0 0x7f81d015ecf0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f81d00bfc90 0x7f81d01f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 --2- 10.20.193.51:0/1454485624 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f81d00bf800 0x7f81d01f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 >> 10.20.193.51:0/1454485624 conn(0x7f81d00cd7b0 msgr2=0x7f81d00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 shutdown_connections 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 -- 10.20.193.51:0/1454485624 wait complete. 2026-02-22T21:40:35.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:35.435+0000 7f81d6205640 1 librados: shutdown 2026-02-22T21:40:38.448 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 10 monclient: get_monmap_and_config 2026-02-22T21:40:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 10 monclient: build_initial_monmap 2026-02-22T21:40:38.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 10 monclient: monmap: 2026-02-22T21:40:38.517 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:38.517019+0000 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:38.517019+0000 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 1 Processor -- start 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 1 -- start start 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.515+0000 7f094f3e4640 10 monclient: init 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient: _add_conns ranks=[2,1,3,5,4,0] 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0948150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f094815c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f09480703b0 0x7f09480c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-a con 0x7f09480703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094815a920 con 0x7f094815c710 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f094815af60 con 0x7f0948150220 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f094815b240 con 0x7f09480703b0 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094f3e4640 10 monclient(hunting): authenticate will time out at 1314.097315s 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094d95a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f09480703b0 0x7f09480c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094d159640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f0948150220 0x7f094815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:33544/0 (socket says 10.20.193.51:33544) 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.516+0000 7f094d159640 1 -- 10.20.193.51:0/2454363076 learned_addr learned my addr 10.20.193.51:0/2454363076 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): get_auth_request con 0x7f09480703b0 auth_method 0 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.517+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 1 -- 10.20.193.51:0/2454363076 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099388682 0 0) 0x7f094815af60 con 0x7f0948150220 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 10 monclient(hunting): _negotiate 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 10 monclient(hunting): my global_id is 4889 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f0937fff640 1 -- 10.20.193.51:0/2454363076 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0930003620 con 0x7f0948150220 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 10 monclient(hunting): handle_auth_done global_id 4932 payload 274 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 1 -- 10.20.193.51:0/2454363076 >> v1:10.20.193.125:6789/0 conn(0x7f0948150220 legacy=0x7f094815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 1 -- 10.20.193.51:0/2454363076 >> v1:10.20.193.68:6789/0 conn(0x7f094815c710 legacy=0x7f094806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 10 monclient: _finish_hunting 0 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 1 monclient: found mon.noname-a 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.518+0000 7f094d95a640 1 -- 10.20.193.51:0/2454363076 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094815b520 con 0x7f09480703b0 2026-02-22T21:40:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.519+0000 7f094d95a640 1 --2- 10.20.193.51:0/2454363076 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09480c73a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0944009dc0 tx=0x7f094402f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c8a57f9ab06a46 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 1 -- 10.20.193.51:0/2454363076 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094403c070 con 0x7f09480703b0 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 10 monclient: dump: 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 10 monclient: _finish_auth 0 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 10 monclient: _check_auth_tickets 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f0937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f094f3e4640 5 monclient: authenticate success, global_id 4932 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.520+0000 7f094f3e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 1 -- 10.20.193.51:0/2454363076 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0944037440 con 0x7f09480703b0 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 1 -- 10.20.193.51:0/2454363076 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440377f0 con 0x7f09480703b0 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f0937fff640 10 monclient: dump: 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 4 set_mon_vals no callback set 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 10 monclient: shutdown 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 -- 10.20.193.51:0/2454363076 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 msgr2=0x7f09480c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 --2- 10.20.193.51:0/2454363076 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09480c73a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0944009dc0 tx=0x7f094402f2f0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 -- 10.20.193.51:0/2454363076 shutdown_connections 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 --2- 10.20.193.51:0/2454363076 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09480c73a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 -- 10.20.193.51:0/2454363076 >> 10.20.193.51:0/2454363076 conn(0x7f09480cd7b0 msgr2=0x7f09480cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 -- 10.20.193.51:0/2454363076 shutdown_connections 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 1 -- 10.20.193.51:0/2454363076 wait complete. 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 10 monclient: build_initial_monmap 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.521+0000 7f094f3e4640 10 monclient: monmap: 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:38.524 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 librados: starting msgr at 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 librados: starting objecter 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 Processor -- start 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 -- start start 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 librados: setting wanted keys 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 librados: calling monclient init 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient: init 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-a con 0x7f09480703b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0948150220 0x7f0948158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-c con 0x7f0948150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f094815c710 0x7f09481f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): picked mon.noname-b con 0x7f094815c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0948150600 con 0x7f09480703b0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f094815bae0 con 0x7f094815c710 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0948150780 con 0x7f0948150220 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.522+0000 7f094f3e4640 10 monclient(hunting): authenticate will time out at 1314.103946s 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0948150220 0x7f0948158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43026/0 (socket says 10.20.193.51:43026) 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 1 -- 10.20.193.51:0/953342877 learned_addr learned my addr 10.20.193.51:0/953342877 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): get_auth_request con 0x7f09480703b0 auth_method 0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f094815c710 0x7f09481f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d159640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): get_auth_request con 0x7f094815c710 auth_method 0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): get_auth_request con 0x7f0948150220 auth_method 0 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:38.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094c958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.523+0000 7f094d95a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 10 monclient(hunting): handle_auth_done global_id 4935 payload 824 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 -- 10.20.193.51:0/953342877 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0948150220 msgr2=0x7f0948158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0948150220 0x7f0948158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 -- 10.20.193.51:0/953342877 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f094815c710 msgr2=0x7f09481f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f094815c710 0x7f09481f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 10 monclient: _finish_hunting 0 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 monclient: found mon.noname-a 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.524+0000 7f094d159640 1 -- 10.20.193.51:0/953342877 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094814ce50 con 0x7f09480703b0 2026-02-22T21:40:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.525+0000 7f094d159640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f093800c700 tx=0x7f093800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.525+0000 7f0935ffb640 10 client.?.objecter ms_handle_connect 0x7f09480703b0 2026-02-22T21:40:38.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.525+0000 7f0935ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938017070 con 0x7f09480703b0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 10 monclient: dump: 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 10 monclient: _finish_auth 0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f0935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f094f3e4640 5 monclient: authenticate success, global_id 4935 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f094f3e4640 10 monclient: _renew_subs 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f094f3e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.526+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0948157970 con 0x7f09480703b0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 client.4935.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 monclient: _renew_subs 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0948157f30 con 0x7f09480703b0 2026-02-22T21:40:38.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 1 librados: init done 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09480c73a0 con 0x7f09480703b0 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0938005bd0 con 0x7f09480703b0 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938007070 con 0x7f09480703b0 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f09377fe640 4 set_mon_vals no callback set 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.527+0000 7f0935ffb640 10 monclient: dump: 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:38.530 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:38.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.636+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f0938007210 con 0x7f09480703b0 2026-02-22T21:40:38.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.636+0000 7f0935ffb640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 0x7f091807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:38.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.636+0000 7f094c958640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 0x7f091807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:38.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.637+0000 7f094c958640 10 monclient: get_auth_request con 0x7f0918078e00 auth_method 0 2026-02-22T21:40:38.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.637+0000 7f0935ffb640 10 client.4935.objecter ms_handle_connect 0x7f0918078e00 2026-02-22T21:40:38.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.637+0000 7f094c958640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 0x7f091807b2c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f093c002410 tx=0x7f093c007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f0938096e80 con 0x7f09480703b0 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 10 client.4935.objecter ms_dispatch2 0x7f0948000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 3 client.4935.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 3 client.4935.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 20 client.4935.objecter dump_active .. 0 homeless 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0938022020 con 0x7f09480703b0 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:38.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.645+0000 7f0935ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:38.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.746+0000 7f094f3e4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:38.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.746+0000 7f094f3e4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:38.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.746+0000 7f094f3e4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:38.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.746+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0948074da0 con 0x7f09480703b0 2026-02-22T21:40:38.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.749+0000 7f0935ffb640 1 -- 10.20.193.51:0/953342877 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f09380614b0 con 0x7f09480703b0 2026-02-22T21:40:38.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.749+0000 7f0935ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:38.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.749+0000 7f0935ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:38.752 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:38.752 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:38.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 10 librados: watch_flush enter 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 10 librados: watch_flush exit 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 20 client.4935.objecter shutdown clearing up homeless session... 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 10 client.4935.objecter successfully canceled tick 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 msgr2=0x7f091807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 0x7f091807b2c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f093c002410 tx=0x7f093c007ef0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 10 monclient: shutdown 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 msgr2=0x7f09481f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f093800c700 tx=0x7f093800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 shutdown_connections 2026-02-22T21:40:38.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0918078e00 0x7f091807b2c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f094815c710 0x7f09481f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0948150220 0x7f0948158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 --2- 10.20.193.51:0/953342877 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09480703b0 0x7f09481f4f20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 >> 10.20.193.51:0/953342877 conn(0x7f09480cd7b0 msgr2=0x7f094806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 shutdown_connections 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 -- 10.20.193.51:0/953342877 wait complete. 2026-02-22T21:40:38.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:38.750+0000 7f094f3e4640 1 librados: shutdown 2026-02-22T21:40:41.517 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:12.671 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.519 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:12.673 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.525 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:12.679 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.545 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:12.661 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.546 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:12.697 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.547 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:12.698 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.553 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:12.669 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.555 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:12.706 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.562 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:12.678 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:40:41.761 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:41.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 10 monclient: get_monmap_and_config 2026-02-22T21:40:41.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 10 monclient: build_initial_monmap 2026-02-22T21:40:41.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 10 monclient: monmap: 2026-02-22T21:40:41.828 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:41.828010+0000 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:41.828010+0000 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 1 Processor -- start 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 1 -- start start 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.826+0000 7f7136dd3640 10 monclient: init 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient: _add_conns ranks=[2,5,3,4,1,0] 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71301527b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f71300bf800 0x7f713006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-c con 0x7f71300bf800 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f71300bfc90 0x7f71300701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-a con 0x7f71300bfc90 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713015ffa0 con 0x7f71301527b0 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f71300be810 con 0x7f71300bfc90 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f7130152b90 con 0x7f71300bf800 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7136dd3640 10 monclient(hunting): authenticate will time out at 1317.408427s 2026-02-22T21:40:41.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7135349640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f71300bfc90 0x7f71300701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.827+0000 7f7127fff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f71300bf800 0x7f713006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f71300bfc90 0x7f71300701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43042/0 (socket says 10.20.193.51:43042) 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 1 -- 10.20.193.51:0/3453955433 learned_addr learned my addr 10.20.193.51:0/3453955433 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): get_auth_request con 0x7f71300bfc90 auth_method 0 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): get_auth_request con 0x7f71300bf800 auth_method 0 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:41.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.828+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 10 monclient(hunting): handle_auth_done global_id 4938 payload 274 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 1 -- 10.20.193.51:0/3453955433 >> v2:10.20.193.125:3300/0 conn(0x7f71300bf800 msgr2=0x7f713006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 1 --2- 10.20.193.51:0/3453955433 >> v2:10.20.193.125:3300/0 conn(0x7f71300bf800 0x7f713006fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 1 -- 10.20.193.51:0/3453955433 >> v1:10.20.193.125:6789/0 conn(0x7f71301527b0 legacy=0x7f71300c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 10 monclient: _finish_hunting 0 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 1 monclient: found mon.noname-a 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.829+0000 7f7135349640 1 -- 10.20.193.51:0/3453955433 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71300bea60 con 0x7f71300bfc90 2026-02-22T21:40:41.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.830+0000 7f7135349640 1 --2- 10.20.193.51:0/3453955433 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300701e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f71300709e0 tx=0x7f712c02f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a56a119b17b9ea8 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 1 -- 10.20.193.51:0/3453955433 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c02fd70 con 0x7f71300bfc90 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: dump: 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:41.834 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: _finish_auth 0 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: _check_auth_tickets 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 5 monclient: authenticate success, global_id 4938 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 1 -- 10.20.193.51:0/3453955433 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712c0373f0 con 0x7f71300bfc90 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 1 -- 10.20.193.51:0/3453955433 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c0377c0 con 0x7f71300bfc90 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 4 set_mon_vals no callback set 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f71277fe640 10 monclient: dump: 2026-02-22T21:40:41.835 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 10 monclient: shutdown 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 -- 10.20.193.51:0/3453955433 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 msgr2=0x7f71300701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 --2- 10.20.193.51:0/3453955433 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300701e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f71300709e0 tx=0x7f712c02f310 comp rx=0 tx=0).stop 2026-02-22T21:40:41.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 -- 10.20.193.51:0/3453955433 shutdown_connections 2026-02-22T21:40:41.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 --2- 10.20.193.51:0/3453955433 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300701e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:41.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 --2- 10.20.193.51:0/3453955433 >> v2:10.20.193.125:3300/0 conn(0x7f71300bf800 0x7f713006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:41.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 -- 10.20.193.51:0/3453955433 >> 10.20.193.51:0/3453955433 conn(0x7f71300cd7b0 msgr2=0x7f71300cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:41.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 -- 10.20.193.51:0/3453955433 shutdown_connections 2026-02-22T21:40:41.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 1 -- 10.20.193.51:0/3453955433 wait complete. 2026-02-22T21:40:41.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 10 monclient: build_initial_monmap 2026-02-22T21:40:41.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.832+0000 7f7136dd3640 10 monclient: monmap: 2026-02-22T21:40:41.840 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 librados: starting msgr at 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 librados: starting objecter 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 Processor -- start 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 -- start start 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 librados: setting wanted keys 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 librados: calling monclient init 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient: init 2026-02-22T21:40:41.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71300bf800 0x7f713015ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-c con 0x7f71300bf800 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-a con 0x7f71300bfc90 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71301527b0 0x7f71300c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): picked mon.noname-b con 0x7f71301527b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f71300bf300 con 0x7f71300bfc90 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f71300bf020 con 0x7f71301527b0 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f713006d710 con 0x7f71300bf800 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7136dd3640 10 monclient(hunting): authenticate will time out at 1317.414631s 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7127fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.833+0000 7f7134b48640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71300bf800 0x7f713015ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71301527b0 0x7f71300c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43048/0 (socket says 10.20.193.51:43048) 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 1 -- 10.20.193.51:0/2987294324 learned_addr learned my addr 10.20.193.51:0/2987294324 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): get_auth_request con 0x7f71300bfc90 auth_method 0 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): get_auth_request con 0x7f71300bf800 auth_method 0 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): get_auth_request con 0x7f71301527b0 auth_method 0 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:41.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7134b48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.834+0000 7f7135349640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 10 monclient(hunting): handle_auth_done global_id 4941 payload 824 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 -- 10.20.193.51:0/2987294324 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71300bf800 msgr2=0x7f713015ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71300bf800 0x7f713015ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 -- 10.20.193.51:0/2987294324 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71301527b0 msgr2=0x7f71300c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71301527b0 0x7f71300c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 monclient: found mon.noname-a 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.835+0000 7f7127fff640 1 -- 10.20.193.51:0/2987294324 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71300c7490 con 0x7f71300bfc90 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.836+0000 7f7127fff640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f711800e890 tx=0x7f711800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.836+0000 7f71257fa640 10 client.?.objecter ms_handle_connect 0x7f71300bfc90 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.836+0000 7f71257fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118017070 con 0x7f71300bfc90 2026-02-22T21:40:41.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 10 monclient: dump: 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 10 monclient: _finish_auth 0 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 10 monclient: _check_auth_tickets 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f71257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f7136dd3640 5 monclient: authenticate success, global_id 4941 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f7136dd3640 10 monclient: _renew_subs 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f7136dd3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.837+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713015dbc0 con 0x7f71300bfc90 2026-02-22T21:40:41.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 client.4941.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 monclient: _renew_subs 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713015e180 con 0x7f71300bfc90 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 1 librados: init done 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71300c9660 con 0x7f71300bfc90 2026-02-22T21:40:41.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7118005a90 con 0x7f71300bfc90 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118021060 con 0x7f71300bfc90 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f7126ffd640 4 set_mon_vals no callback set 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.838+0000 7f71257fa640 10 monclient: dump: 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:41.847 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:41.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7118024060 con 0x7f71300bfc90 2026-02-22T21:40:41.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f71257fa640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 0x7f710007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:41.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f7134b48640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 0x7f710007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:41.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f7134b48640 10 monclient: get_auth_request con 0x7f7100078e00 auth_method 0 2026-02-22T21:40:41.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f7134b48640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 0x7f710007b2c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7120004680 tx=0x7f7120009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:41.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.949+0000 7f71257fa640 10 client.4941.objecter ms_handle_connect 0x7f7100078e00 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f711805e060 con 0x7f71300bfc90 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 10 client.4941.objecter ms_dispatch2 0x7f7130000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 3 client.4941.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 3 client.4941.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 20 client.4941.objecter dump_active .. 0 homeless 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7118062020 con 0x7f71300bfc90 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:41.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:41.957+0000 7f71257fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.056+0000 7f7136dd3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.056+0000 7f7136dd3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.056+0000 7f7136dd3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:42.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.056+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f71301596e0 con 0x7f71300bfc90 2026-02-22T21:40:42.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.060+0000 7f71257fa640 1 -- 10.20.193.51:0/2987294324 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f7118066320 con 0x7f71300bfc90 2026-02-22T21:40:42.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.060+0000 7f71257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:42.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.060+0000 7f71257fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:42.062 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:42.062 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 10 librados: watch_flush enter 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 10 librados: watch_flush exit 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 20 client.4941.objecter shutdown clearing up homeless session... 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 10 client.4941.objecter successfully canceled tick 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 msgr2=0x7f710007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 0x7f710007b2c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7120004680 tx=0x7f7120009190 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 10 monclient: shutdown 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 msgr2=0x7f71300c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f711800e890 tx=0x7f711800ed60 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 shutdown_connections 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7100078e00 0x7f710007b2c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f71301527b0 0x7f71300c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f71300bfc90 0x7f71300c6a10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 --2- 10.20.193.51:0/2987294324 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f71300bf800 0x7f713015ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 >> 10.20.193.51:0/2987294324 conn(0x7f71300cd7b0 msgr2=0x7f71300c8290 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 shutdown_connections 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 -- 10.20.193.51:0/2987294324 wait complete. 2026-02-22T21:40:42.064 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:42.061+0000 7f7136dd3640 1 librados: shutdown 2026-02-22T21:40:45.072 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.138+0000 7f74f4d7e640 10 monclient: get_monmap_and_config 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.138+0000 7f74f4d7e640 10 monclient: build_initial_monmap 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.138+0000 7f74f4d7e640 10 monclient: monmap: 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:45.140502+0000 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:45.140502+0000 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:45.141 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 Processor -- start 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 -- start start 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient: init 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient: _add_conns ranks=[3,5,0,4,2,1] 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 0x7f74f00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f74f0150600 0x7f74f015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-c con 0x7f74f0150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74f015c540 addr v1:10.20.193.51:6789/0 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74f015c9b0 con 0x7f74f015c540 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f74f015a920 con 0x7f74f0150220 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f74f015af60 con 0x7f74f0150600 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.139+0000 7f74f4d7e640 10 monclient(hunting): authenticate will time out at 1320.720914s 2026-02-22T21:40:45.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 0x7f74f00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f74f0150600 0x7f74f015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 0x7f74f00c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43062/0 (socket says 10.20.193.51:43062) 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 1 -- 10.20.193.51:0/4198653340 learned_addr learned my addr 10.20.193.51:0/4198653340 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 10 monclient(hunting): get_auth_request con 0x7f74f0150220 auth_method 0 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74ee575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): get_auth_request con 0x7f74f0150600 auth_method 0 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.140+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:45.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 10 monclient(hunting): handle_auth_done global_id 4901 payload 274 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 1 -- 10.20.193.51:0/4198653340 >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 msgr2=0x7f74f00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 1 --2- 10.20.193.51:0/4198653340 >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 0x7f74f00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 1 -- 10.20.193.51:0/4198653340 >> v1:10.20.193.51:6789/0 conn(0x7f74f015c540 legacy=0x7f74f006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 10 monclient: _finish_hunting 0 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 1 monclient: found mon.noname-c 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.141+0000 7f74edd74640 1 -- 10.20.193.51:0/4198653340 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f015b240 con 0x7f74f0150600 2026-02-22T21:40:45.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.142+0000 7f74edd74640 1 --2- 10.20.193.51:0/4198653340 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150600 0x7f74f015c000 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f74f0074940 tx=0x7f74d802eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd6b4b92c8b3330b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 1 -- 10.20.193.51:0/4198653340 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d803c070 con 0x7f74f0150600 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: dump: 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: _finish_auth 0 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: _check_auth_tickets 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 1 -- 10.20.193.51:0/4198653340 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d802fb80 con 0x7f74f0150600 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 1 -- 10.20.193.51:0/4198653340 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8037440 con 0x7f74f0150600 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 5 monclient: authenticate success, global_id 4901 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 4 set_mon_vals no callback set 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74ed573640 10 monclient: dump: 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:45.147 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 10 monclient: shutdown 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 -- 10.20.193.51:0/4198653340 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150600 msgr2=0x7f74f015c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/4198653340 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150600 0x7f74f015c000 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f74f0074940 tx=0x7f74d802eeb0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 -- 10.20.193.51:0/4198653340 shutdown_connections 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/4198653340 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150600 0x7f74f015c000 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/4198653340 >> v2:10.20.193.51:3300/0 conn(0x7f74f0150220 0x7f74f00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 -- 10.20.193.51:0/4198653340 >> 10.20.193.51:0/4198653340 conn(0x7f74f00cd7b0 msgr2=0x7f74f00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 -- 10.20.193.51:0/4198653340 shutdown_connections 2026-02-22T21:40:45.148 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.144+0000 7f74f4d7e640 1 -- 10.20.193.51:0/4198653340 wait complete. 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 10 monclient: build_initial_monmap 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 10 monclient: monmap: 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 librados: starting msgr at 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 librados: starting objecter 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 Processor -- start 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 -- start start 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 librados: setting wanted keys 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 1 librados: calling monclient init 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.145+0000 7f74f4d7e640 10 monclient: init 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150220 0x7f74f006d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-c con 0x7f74f0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f74f015c540 0x7f74f006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): picked mon.noname-b con 0x7f74f015c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f74f015bae0 con 0x7f74f0150600 2026-02-22T21:40:45.149 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f74f015b800 con 0x7f74f015c540 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f74f00703b0 con 0x7f74f0150220 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74f4d7e640 10 monclient(hunting): authenticate will time out at 1320.727401s 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74ee575640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150220 0x7f74f006d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43068/0 (socket says 10.20.193.51:43068) 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 1 -- 10.20.193.51:0/3101597424 learned_addr learned my addr 10.20.193.51:0/3101597424 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 10 monclient(hunting): get_auth_request con 0x7f74f0150600 auth_method 0 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74edd74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74ee575640 10 monclient(hunting): get_auth_request con 0x7f74f0150220 auth_method 0 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74ee575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74ee575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74ee575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.146+0000 7f74eed76640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f74f015c540 0x7f74f006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:45.150 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74edd74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74ee575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): get_auth_request con 0x7f74f015c540 auth_method 0 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.147+0000 7f74eed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 10 monclient(hunting): handle_auth_done global_id 4947 payload 824 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 -- 10.20.193.51:0/3101597424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150220 msgr2=0x7f74f006d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150220 0x7f74f006d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 -- 10.20.193.51:0/3101597424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f74f015c540 msgr2=0x7f74f006e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f74f015c540 0x7f74f006e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 10 monclient: _finish_hunting 0 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 monclient: found mon.noname-a 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.148+0000 7f74edd74640 1 -- 10.20.193.51:0/3101597424 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f006ec10 con 0x7f74f0150600 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.149+0000 7f74edd74640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009130 tx=0x7f74d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.149+0000 7f74d6ffd640 10 client.?.objecter ms_handle_connect 0x7f74f0150600 2026-02-22T21:40:45.151 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.149+0000 7f74d6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d803c040 con 0x7f74f0150600 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: dump: 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:45.153 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d8042610 con 0x7f74f0150600 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d80429e0 con 0x7f74f0150600 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74d6ffd640 10 monclient: dump: 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74ecd72640 4 set_mon_vals no callback set 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 5 monclient: authenticate success, global_id 4947 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _renew_subs 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f006eef0 con 0x7f74f0150600 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 client.4947.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _renew_subs 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f0072ff0 con 0x7f74f0150600 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 1 librados: init done 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:45.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:45.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:45.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.151+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f006f410 con 0x7f74f0150600 2026-02-22T21:40:45.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.163+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f74d804b450 con 0x7f74f0150600 2026-02-22T21:40:45.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.163+0000 7f74d6ffd640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 0x7f74bc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:45.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.163+0000 7f74ee575640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 0x7f74bc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:45.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.163+0000 7f74ee575640 10 monclient: get_auth_request con 0x7f74bc078e50 auth_method 0 2026-02-22T21:40:45.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74ee575640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 0x7f74bc07b310 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f74e4006dc0 tx=0x7f74e4009340 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:45.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 10 client.4947.objecter ms_handle_connect 0x7f74bc078e50 2026-02-22T21:40:45.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f74d802fb00 con 0x7f74f0150600 2026-02-22T21:40:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 10 client.4947.objecter ms_dispatch2 0x7f74f0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 3 client.4947.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 3 client.4947.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.164+0000 7f74d6ffd640 20 client.4947.objecter dump_active .. 0 homeless 2026-02-22T21:40:45.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.267+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f74f006f410 con 0x7f74f0150600 2026-02-22T21:40:45.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.267+0000 7f74d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:45.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.267+0000 7f74d6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:45.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.366+0000 7f74f4d7e640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:45.369 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.366+0000 7f74f4d7e640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:45.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.366+0000 7f74f4d7e640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:45.370 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.366+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f74f015c000 con 0x7f74f0150600 2026-02-22T21:40:45.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.369+0000 7f74d6ffd640 1 -- 10.20.193.51:0/3101597424 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f74d8086880 con 0x7f74f0150600 2026-02-22T21:40:45.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.369+0000 7f74d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:45.372 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.369+0000 7f74d6ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:45.373 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:45.373 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:45.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 10 librados: watch_flush enter 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 10 librados: watch_flush exit 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 20 client.4947.objecter shutdown clearing up homeless session... 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 10 client.4947.objecter successfully canceled tick 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 msgr2=0x7f74bc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 0x7f74bc07b310 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f74e4006dc0 tx=0x7f74e4009340 comp rx=0 tx=0).stop 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 10 monclient: shutdown 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 msgr2=0x7f74f01f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009130 tx=0x7f74d80047c0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 shutdown_connections 2026-02-22T21:40:45.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f74bc078e50 0x7f74bc07b310 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f74f015c540 0x7f74f006e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f74f0150600 0x7f74f01f6f10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 --2- 10.20.193.51:0/3101597424 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f74f0150220 0x7f74f006d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 >> 10.20.193.51:0/3101597424 conn(0x7f74f00cd7b0 msgr2=0x7f74f00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 shutdown_connections 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 -- 10.20.193.51:0/3101597424 wait complete. 2026-02-22T21:40:45.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:45.370+0000 7f74f4d7e640 1 librados: shutdown 2026-02-22T21:40:48.382 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:48.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.447+0000 7f523b12c640 10 monclient: get_monmap_and_config 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.447+0000 7f523b12c640 10 monclient: build_initial_monmap 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.447+0000 7f523b12c640 10 monclient: monmap: 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:48.449559+0000 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:48.449559+0000 2026-02-22T21:40:48.451 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 Processor -- start 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 -- start start 2026-02-22T21:40:48.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient: init 2026-02-22T21:40:48.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:48.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient: _add_conns ranks=[2,1,3,4,5,0] 2026-02-22T21:40:48.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5234150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:48.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f523415c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f52340703b0 0x7f52340c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-a con 0x7f52340703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523415a920 con 0x7f523415c710 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523415af60 con 0x7f5234150220 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f523415b240 con 0x7f52340703b0 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.448+0000 7f523b12c640 10 monclient(hunting): authenticate will time out at 1324.029963s 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f5238ea1640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f5234150220 0x7f523415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:59630/0 (socket says 10.20.193.51:59630) 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f5238ea1640 1 -- 10.20.193.51:0/2728833435 learned_addr learned my addr 10.20.193.51:0/2728833435 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:48.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 1 --2- 10.20.193.51:0/2728833435 >> v2:10.20.193.51:3300/0 conn(0x7f52340703b0 0x7f52340c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): get_auth_request con 0x7f52340703b0 auth_method 0 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.449+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2910072497 0 0) 0x7f523415af60 con 0x7f5234150220 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): my global_id is 4907 2026-02-22T21:40:48.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5220003620 con 0x7f5234150220 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692818379 0 0) 0x7f523415a920 con 0x7f523415c710 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 10 monclient(hunting): my global_id is 4939 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f523415af60 con 0x7f523415c710 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.450+0000 7f52396a2640 10 monclient(hunting): handle_auth_done global_id 4950 payload 274 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 1 -- 10.20.193.51:0/2728833435 >> v1:10.20.193.125:6789/0 conn(0x7f5234150220 legacy=0x7f523415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 1 -- 10.20.193.51:0/2728833435 >> v1:10.20.193.68:6789/0 conn(0x7f523415c710 legacy=0x7f523406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 10 monclient: _finish_hunting 0 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 1 monclient: found mon.noname-a 2026-02-22T21:40:48.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:48.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 1 -- 10.20.193.51:0/2728833435 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523415b520 con 0x7f52340703b0 2026-02-22T21:40:48.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.451+0000 7f52396a2640 1 --2- 10.20.193.51:0/2728833435 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f52340703b0 0x7f52340c73a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f523002f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8e90ddd35a4bda server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:48.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523003c070 con 0x7f52340703b0 2026-02-22T21:40:48.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: dump: 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:48.466 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5230037440 con 0x7f52340703b0 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 1 -- 10.20.193.51:0/2728833435 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230037810 con 0x7f52340703b0 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:48.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f522b7fe640 10 monclient: dump: 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f523b12c640 5 monclient: authenticate success, global_id 4950 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f523b12c640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.453+0000 7f523b12c640 4 set_mon_vals no callback set 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 10 monclient: shutdown 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- 10.20.193.51:0/2728833435 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f52340703b0 msgr2=0x7f52340c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 --2- 10.20.193.51:0/2728833435 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f52340703b0 0x7f52340c73a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f523002f2f0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- 10.20.193.51:0/2728833435 shutdown_connections 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 --2- 10.20.193.51:0/2728833435 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f52340703b0 0x7f52340c73a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- 10.20.193.51:0/2728833435 >> 10.20.193.51:0/2728833435 conn(0x7f52340cd7b0 msgr2=0x7f52340cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- 10.20.193.51:0/2728833435 shutdown_connections 2026-02-22T21:40:48.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- 10.20.193.51:0/2728833435 wait complete. 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 10 monclient: build_initial_monmap 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 10 monclient: monmap: 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:48.470 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 librados: starting msgr at 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 librados: starting objecter 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 Processor -- start 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 -- start start 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 librados: setting wanted keys 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 1 librados: calling monclient init 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.454+0000 7f523b12c640 10 monclient: init 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f52340703b0 0x7f523406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-c con 0x7f52340703b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:48.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-a con 0x7f5234150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f523415c710 0x7f523414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): picked mon.noname-b con 0x7f523415c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f5234150600 con 0x7f5234150220 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f523415bae0 con 0x7f523415c710 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5234150780 con 0x7f52340703b0 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f523b12c640 10 monclient(hunting): authenticate will time out at 1324.036314s 2026-02-22T21:40:48.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f5238ea1640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f52340703b0 0x7f523406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f52396a2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f523415c710 0x7f523414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:59384/0 (socket says 10.20.193.51:59384) 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 1 -- 10.20.193.51:0/1422644861 learned_addr learned my addr 10.20.193.51:0/1422644861 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): get_auth_request con 0x7f5234150220 auth_method 0 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f5238ea1640 10 monclient(hunting): get_auth_request con 0x7f52340703b0 auth_method 0 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f5238ea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:48.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f5238ea1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f5238ea1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f522bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f52396a2640 10 monclient(hunting): get_auth_request con 0x7f523415c710 auth_method 0 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f52396a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f52396a2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.455+0000 7f52396a2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f5238ea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f5238ea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f5238ea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.456+0000 7f52396a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 10 monclient(hunting): handle_auth_done global_id 4953 payload 824 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 -- 10.20.193.51:0/1422644861 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f52340703b0 msgr2=0x7f523406e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f52340703b0 0x7f523406e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 -- 10.20.193.51:0/1422644861 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f523415c710 msgr2=0x7f523414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f523415c710 0x7f523414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 monclient: found mon.noname-a 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.457+0000 7f522bfff640 1 -- 10.20.193.51:0/1422644861 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523406d350 con 0x7f5234150220 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.458+0000 7f522bfff640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f521c00c700 tx=0x7f521c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.458+0000 7f52297fa640 10 client.?.objecter ms_handle_connect 0x7f5234150220 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.458+0000 7f52297fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0052d0 con 0x7f5234150220 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: dump: 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:48.476 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: _finish_auth 0 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: _check_auth_tickets 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c005490 con 0x7f5234150220 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 5 monclient: authenticate success, global_id 4953 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0038d0 con 0x7f5234150220 2026-02-22T21:40:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f52297fa640 10 monclient: dump: 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:48.478 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f522affd640 4 set_mon_vals no callback set 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 10 monclient: _renew_subs 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523406d630 con 0x7f5234150220 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 10 client.4953.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 10 monclient: _renew_subs 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f523406dbf0 con 0x7f5234150220 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.459+0000 7f523b12c640 1 librados: init done 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.460+0000 7f523b12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.460+0000 7f523b12c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.460+0000 7f523b12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.460+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52340c73a0 con 0x7f5234150220 2026-02-22T21:40:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.471+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f521c003a70 con 0x7f5234150220 2026-02-22T21:40:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.471+0000 7f52297fa640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 0x7f520407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f5238ea1640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 0x7f520407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f5238ea1640 10 monclient: get_auth_request con 0x7f5204078e00 auth_method 0 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f5238ea1640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 0x7f520407b2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5224007b60 tx=0x7f5224007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f52297fa640 10 client.4953.objecter ms_handle_connect 0x7f5204078e00 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f521c05e090 con 0x7f5234150220 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f52297fa640 10 client.4953.objecter ms_dispatch2 0x7f5234000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f52297fa640 3 client.4953.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.473+0000 7f52297fa640 3 client.4953.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:48.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.474+0000 7f52297fa640 20 client.4953.objecter dump_active .. 0 homeless 2026-02-22T21:40:48.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.574+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f521c062050 con 0x7f5234150220 2026-02-22T21:40:48.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.574+0000 7f52297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:48.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.574+0000 7f52297fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:48.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.674+0000 7f523b12c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:48.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.674+0000 7f523b12c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:48.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.674+0000 7f523b12c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:48.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.674+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f52340c60a0 con 0x7f5234150220 2026-02-22T21:40:48.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.676+0000 7f52297fa640 1 -- 10.20.193.51:0/1422644861 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f521c066350 con 0x7f5234150220 2026-02-22T21:40:48.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.676+0000 7f52297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:48.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.676+0000 7f52297fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:48.678 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:48.678 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:48.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 10 librados: watch_flush enter 2026-02-22T21:40:48.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 10 librados: watch_flush exit 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 20 client.4953.objecter shutdown clearing up homeless session... 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 10 client.4953.objecter successfully canceled tick 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 msgr2=0x7f520407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 0x7f520407b2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5224007b60 tx=0x7f5224007ab0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 10 monclient: shutdown 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 msgr2=0x7f52341f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f521c00c700 tx=0x7f521c00cbd0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 shutdown_connections 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5204078e00 0x7f520407b2c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f523415c710 0x7f523414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5234150220 0x7f52341f50a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 --2- 10.20.193.51:0/1422644861 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f52340703b0 0x7f523406e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 >> 10.20.193.51:0/1422644861 conn(0x7f52340cd7b0 msgr2=0x7f52340cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 shutdown_connections 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 -- 10.20.193.51:0/1422644861 wait complete. 2026-02-22T21:40:48.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:48.677+0000 7f523b12c640 1 librados: shutdown 2026-02-22T21:40:51.688 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:51.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.755+0000 7ff4f4340640 10 monclient: get_monmap_and_config 2026-02-22T21:40:51.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.755+0000 7ff4f4340640 10 monclient: build_initial_monmap 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.755+0000 7ff4f4340640 10 monclient: monmap: 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:51.756356+0000 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:51.756356+0000 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:51.758 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 Processor -- start 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 -- start start 2026-02-22T21:40:51.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient: init 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient: _add_conns ranks=[1,2,5,0,3,4] 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7ff4ec150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4ec15c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff4ec0703b0 0x7ff4ec0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:51.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-c con 0x7ff4ec0703b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4ec15a920 con 0x7ff4ec150220 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4ec15af60 con 0x7ff4ec15c710 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ff4ec15b240 con 0x7ff4ec0703b0 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.756+0000 7ff4f4340640 10 monclient(hunting): authenticate will time out at 1327.336665s 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f20b5640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7ff4ec150220 0x7ff4ec15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:40524/0 (socket says 10.20.193.51:40524) 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f20b5640 1 -- 10.20.193.51:0/233446243 learned_addr learned my addr 10.20.193.51:0/233446243 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f28b6640 1 --2- 10.20.193.51:0/233446243 >> v2:10.20.193.125:3300/0 conn(0x7ff4ec0703b0 0x7ff4ec0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f28b6640 10 monclient(hunting): get_auth_request con 0x7ff4ec0703b0 auth_method 0 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f28b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:51.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f28b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.757+0000 7ff4f28b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.758+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.758+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.758+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023338575 0 0) 0x7ff4ec15af60 con 0x7ff4ec15c710 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _negotiate 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): my global_id is 4916 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d4003620 con 0x7ff4ec15c710 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018987302 0 0) 0x7ff4ec15a920 con 0x7ff4ec150220 2026-02-22T21:40:51.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _negotiate 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 10 monclient(hunting): my global_id is 4945 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4ec15af60 con 0x7ff4ec150220 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_done global_id 4913 payload 274 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 1 -- 10.20.193.51:0/233446243 >> v1:10.20.193.125:6789/0 conn(0x7ff4ec15c710 legacy=0x7ff4ec06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 1 -- 10.20.193.51:0/233446243 >> v1:10.20.193.68:6789/0 conn(0x7ff4ec150220 legacy=0x7ff4ec15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 10 monclient: _finish_hunting 0 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 1 monclient: found mon.noname-c 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.759+0000 7ff4f28b6640 1 -- 10.20.193.51:0/233446243 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec15b520 con 0x7ff4ec0703b0 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.762+0000 7ff4f28b6640 1 --2- 10.20.193.51:0/233446243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec0c73a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009dc0 tx=0x7ff4e802f2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c1babe6a933774 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e803c070 con 0x7ff4ec0703b0 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: dump: 2026-02-22T21:40:51.771 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: _finish_auth 0 2026-02-22T21:40:51.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: _check_auth_tickets 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 5 monclient: authenticate success, global_id 4913 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e8037440 con 0x7ff4ec0703b0 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 1 -- 10.20.193.51:0/233446243 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80377f0 con 0x7ff4ec0703b0 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f10b3640 10 monclient: dump: 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:51.773 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 4 set_mon_vals no callback set 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 10 monclient: shutdown 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:51.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 -- 10.20.193.51:0/233446243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec0703b0 msgr2=0x7ff4ec0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 --2- 10.20.193.51:0/233446243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec0c73a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8009dc0 tx=0x7ff4e802f2f0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 -- 10.20.193.51:0/233446243 shutdown_connections 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 --2- 10.20.193.51:0/233446243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec0c73a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 -- 10.20.193.51:0/233446243 >> 10.20.193.51:0/233446243 conn(0x7ff4ec0cd7b0 msgr2=0x7ff4ec0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 -- 10.20.193.51:0/233446243 shutdown_connections 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.764+0000 7ff4f4340640 1 -- 10.20.193.51:0/233446243 wait complete. 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 10 monclient: build_initial_monmap 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 10 monclient: monmap: 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:51.775 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 librados: starting msgr at 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 librados: starting objecter 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 Processor -- start 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 -- start start 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 librados: setting wanted keys 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 1 librados: calling monclient init 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.765+0000 7ff4f4340640 10 monclient: init 2026-02-22T21:40:51.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-b con 0x7ff4ec0703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-c con 0x7ff4ec150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4ec15c710 0x7ff4ec14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): picked mon.noname-a con 0x7ff4ec15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec150600 con 0x7ff4ec15c710 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec15bae0 con 0x7ff4ec0703b0 2026-02-22T21:40:51.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec150780 con 0x7ff4ec150220 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f4340640 10 monclient(hunting): authenticate will time out at 1327.346149s 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f28b6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4ec15c710 0x7ff4ec14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:41238/0 (socket says 10.20.193.51:41238) 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 1 -- 10.20.193.51:0/651812317 learned_addr learned my addr 10.20.193.51:0/651812317 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): get_auth_request con 0x7ff4ec150220 auth_method 0 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): get_auth_request con 0x7ff4ec0703b0 auth_method 0 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f28b6640 10 monclient(hunting): get_auth_request con 0x7ff4ec15c710 auth_method 0 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f28b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f28b6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f28b6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f20b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:51.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.766+0000 7ff4f18b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.767+0000 7ff4f18b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.767+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.767+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.767+0000 7ff4f28b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 10 monclient(hunting): handle_auth_done global_id 4919 payload 824 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 -- 10.20.193.51:0/651812317 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4ec0703b0 msgr2=0x7ff4ec1f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec1f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 -- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4ec15c710 msgr2=0x7ff4ec14c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4ec15c710 0x7ff4ec14c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 10 monclient: _finish_hunting 0 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 monclient: found mon.noname-c 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:51.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.768+0000 7ff4f18b4640 1 -- 10.20.193.51:0/651812317 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec1f49b0 con 0x7ff4ec150220 2026-02-22T21:40:51.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.769+0000 7ff4f18b4640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c730 tx=0x7ff4e000cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:51.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.769+0000 7ff4d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff4ec150220 2026-02-22T21:40:51.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.769+0000 7ff4d2ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e00052d0 con 0x7ff4ec150220 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 10 monclient: dump: 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.783 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 5 monclient: authenticate success, global_id 4919 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _renew_subs 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ec0c2dc0 con 0x7ff4ec150220 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 client.4919.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _renew_subs 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ec0c3380 con 0x7ff4ec150220 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 1 librados: init done 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.770+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ec0c73a0 con 0x7ff4ec150220 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e0005470 con 0x7ff4ec150220 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:51.785 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0003870 con 0x7ff4ec150220 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4d2ffd640 10 monclient: dump: 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:51.786 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:51.787 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:51.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.771+0000 7ff4f08b2640 4 set_mon_vals no callback set 2026-02-22T21:40:51.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.875+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff4e0003a10 con 0x7ff4ec150220 2026-02-22T21:40:51.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.876+0000 7ff4d2ffd640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 0x7ff4bc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:51.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.876+0000 7ff4f20b5640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 0x7ff4bc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:51.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.876+0000 7ff4f20b5640 10 monclient: get_auth_request con 0x7ff4bc078e00 auth_method 0 2026-02-22T21:40:51.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.876+0000 7ff4f20b5640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 0x7ff4bc07b2c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc005980 tx=0x7ff4dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:51.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.876+0000 7ff4d2ffd640 10 client.4919.objecter ms_handle_connect 0x7ff4bc078e00 2026-02-22T21:40:51.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.878+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff4e005e0b0 con 0x7ff4ec150220 2026-02-22T21:40:51.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.878+0000 7ff4d2ffd640 10 client.4919.objecter ms_dispatch2 0x7ff4ec000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:51.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.878+0000 7ff4d2ffd640 3 client.4919.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:51.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.878+0000 7ff4d2ffd640 3 client.4919.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:51.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.878+0000 7ff4d2ffd640 20 client.4919.objecter dump_active .. 0 homeless 2026-02-22T21:40:51.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.885+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff4e0062070 con 0x7ff4ec150220 2026-02-22T21:40:51.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.885+0000 7ff4d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:51.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.885+0000 7ff4d2ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:51.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.983+0000 7ff4f4340640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:51.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.983+0000 7ff4f4340640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:51.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.983+0000 7ff4f4340640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:51.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.983+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff4ec074da0 con 0x7ff4ec150220 2026-02-22T21:40:51.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.987+0000 7ff4d2ffd640 1 -- 10.20.193.51:0/651812317 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff4e0066370 con 0x7ff4ec150220 2026-02-22T21:40:51.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.987+0000 7ff4d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:51.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.987+0000 7ff4d2ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:51.989 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:51.990 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:51.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 10 librados: watch_flush enter 2026-02-22T21:40:51.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 10 librados: watch_flush exit 2026-02-22T21:40:51.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 20 client.4919.objecter shutdown clearing up homeless session... 2026-02-22T21:40:51.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 10 client.4919.objecter successfully canceled tick 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 msgr2=0x7ff4bc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 0x7ff4bc07b2c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc005980 tx=0x7ff4dc009040 comp rx=0 tx=0).stop 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 10 monclient: shutdown 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 msgr2=0x7ff4ec0c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c730 tx=0x7ff4e000cc00 comp rx=0 tx=0).stop 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 shutdown_connections 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4bc078e00 0x7ff4bc07b2c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.992 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4ec15c710 0x7ff4ec14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4ec150220 0x7ff4ec0c10d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 --2- 10.20.193.51:0/651812317 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4ec0703b0 0x7ff4ec1f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 >> 10.20.193.51:0/651812317 conn(0x7ff4ec0cd7b0 msgr2=0x7ff4ec0cbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 shutdown_connections 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 -- 10.20.193.51:0/651812317 wait complete. 2026-02-22T21:40:51.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:51.988+0000 7ff4f4340640 1 librados: shutdown 2026-02-22T21:40:54.999 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.068+0000 7fe7cfad3640 10 monclient: get_monmap_and_config 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.068+0000 7fe7cfad3640 10 monclient: build_initial_monmap 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.068+0000 7fe7cfad3640 10 monclient: monmap: 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:55.070 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:55.069245+0000 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:55.069245+0000 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:55.071 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 Processor -- start 2026-02-22T21:40:55.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 -- start start 2026-02-22T21:40:55.074 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient: init 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient: _add_conns ranks=[5,4,2,1,0,3] 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe7c8152720 0x7fe7c80cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c8152720 addr v2:10.20.193.125:3300/0 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe7c8152b00 0x7fe7c815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c8152b00 addr v2:10.20.193.68:3300/0 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7c815f3e0 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c80ccbf0 con 0x7fe7c815f3e0 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c80cdae0 con 0x7fe7c8152b00 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:55.075 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c80cced0 con 0x7fe7c8152720 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.069+0000 7fe7cfad3640 10 monclient(hunting): authenticate will time out at 1330.649700s 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe7c8152b00 0x7fe7c815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd848640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe7c8152720 0x7fe7c80cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7ce049640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fe7c815f3e0 0x7fe7c815f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:59652/0 (socket says 10.20.193.51:59652) 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7ce049640 1 -- 10.20.193.51:0/88300854 learned_addr learned my addr 10.20.193.51:0/88300854 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): get_auth_request con 0x7fe7c8152b00 auth_method 0 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd848640 10 monclient(hunting): get_auth_request con 0x7fe7c8152720 auth_method 0 2026-02-22T21:40:55.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd848640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd848640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd848640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.070+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.071+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.071+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.071+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 10 monclient(hunting): handle_auth_done global_id 4951 payload 274 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 1 -- 10.20.193.51:0/88300854 >> v2:10.20.193.125:3300/0 conn(0x7fe7c8152720 msgr2=0x7fe7c80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 1 --2- 10.20.193.51:0/88300854 >> v2:10.20.193.125:3300/0 conn(0x7fe7c8152720 0x7fe7c80cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 1 -- 10.20.193.51:0/88300854 >> v1:10.20.193.125:6789/0 conn(0x7fe7c815f3e0 legacy=0x7fe7c815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.077 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 10 monclient: _finish_hunting 0 2026-02-22T21:40:55.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 1 monclient: found mon.noname-b 2026-02-22T21:40:55.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:55.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:40:55.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.072+0000 7fe7cd047640 1 -- 10.20.193.51:0/88300854 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c80cd1b0 con 0x7fe7c8152b00 2026-02-22T21:40:55.078 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.073+0000 7fe7cd047640 1 --2- 10.20.193.51:0/88300854 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152b00 0x7fe7c815d2e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe7c808eed0 tx=0x7fe7bc02ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac654349a6d8e91 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 1 -- 10.20.193.51:0/88300854 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc03c070 con 0x7fe7c8152b00 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: dump: 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:55.085 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: _finish_auth 0 2026-02-22T21:40:55.086 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: _check_auth_tickets 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 1 -- 10.20.193.51:0/88300854 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7bc02faf0 con 0x7fe7c8152b00 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 5 monclient: authenticate success, global_id 4951 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 1 -- 10.20.193.51:0/88300854 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc037440 con 0x7fe7c8152b00 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cc846640 10 monclient: dump: 2026-02-22T21:40:55.087 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:55.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 4 set_mon_vals no callback set 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 10 monclient: shutdown 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 -- 10.20.193.51:0/88300854 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152b00 msgr2=0x7fe7c815d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/88300854 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152b00 0x7fe7c815d2e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe7c808eed0 tx=0x7fe7bc02ede0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 -- 10.20.193.51:0/88300854 shutdown_connections 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/88300854 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152b00 0x7fe7c815d2e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/88300854 >> v2:10.20.193.125:3300/0 conn(0x7fe7c8152720 0x7fe7c80cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 -- 10.20.193.51:0/88300854 >> 10.20.193.51:0/88300854 conn(0x7fe7c80d1b50 msgr2=0x7fe7c80d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 -- 10.20.193.51:0/88300854 shutdown_connections 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.075+0000 7fe7cfad3640 1 -- 10.20.193.51:0/88300854 wait complete. 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient: build_initial_monmap 2026-02-22T21:40:55.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient: monmap: 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 librados: starting msgr at 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 librados: starting objecter 2026-02-22T21:40:55.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 Processor -- start 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 -- start start 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 librados: setting wanted keys 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 librados: calling monclient init 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient: init 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152720 0x7fe7c8156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c8152720 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe7c8152b00 0x7fe7c8074210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c8152b00 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c815f3e0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c815d820 con 0x7fe7c8152b00 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c80cd770 con 0x7fe7c8152720 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c815d9a0 con 0x7fe7c815f3e0 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.076+0000 7fe7cfad3640 10 monclient(hunting): authenticate will time out at 1330.656825s 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152720 0x7fe7c8156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe7c8152b00 0x7fe7c8074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:41248/0 (socket says 10.20.193.51:41248) 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 1 -- 10.20.193.51:0/1461820829 learned_addr learned my addr 10.20.193.51:0/1461820829 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): get_auth_request con 0x7fe7c815f3e0 auth_method 0 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): get_auth_request con 0x7fe7c8152720 auth_method 0 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): get_auth_request con 0x7fe7c8152b00 auth_method 0 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:55.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7ce049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd848640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.077+0000 7fe7cd047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.078+0000 7fe7ce049640 10 monclient(hunting): handle_auth_done global_id 4928 payload 824 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.078+0000 7fe7ce049640 1 -- 10.20.193.51:0/1461820829 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152720 msgr2=0x7fe7c8156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.078+0000 7fe7ce049640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152720 0x7fe7c8156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 1 -- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe7c8152b00 msgr2=0x7fe7c8074210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe7c8152b00 0x7fe7c8074210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 10 monclient: _finish_hunting 0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 1 monclient: found mon.noname-c 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 1 -- 10.20.193.51:0/1461820829 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c81f3af0 con 0x7fe7c815f3e0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7ce049640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400cb10 tx=0x7fe7c400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.079+0000 7fe7b67fc640 10 client.?.objecter ms_handle_connect 0x7fe7c815f3e0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.080+0000 7fe7b67fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4017070 con 0x7fe7c815f3e0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:55.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: dump: 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: _finish_auth 0 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c400ac50 con 0x7fe7c815f3e0 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4012600 con 0x7fe7c815f3e0 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b7fff640 4 set_mon_vals no callback set 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7b67fc640 10 monclient: dump: 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:55.097 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 5 monclient: authenticate success, global_id 4928 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 10 monclient: _renew_subs 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c81f3d90 con 0x7fe7c815f3e0 2026-02-22T21:40:55.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 10 client.4928.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 10 monclient: _renew_subs 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c81f4350 con 0x7fe7c815f3e0 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.081+0000 7fe7cfad3640 1 librados: init done 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.082+0000 7fe7cfad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.082+0000 7fe7cfad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.082+0000 7fe7cfad3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:55.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.082+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c80cdf70 con 0x7fe7c815f3e0 2026-02-22T21:40:55.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.093+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe7c40113a0 con 0x7fe7c815f3e0 2026-02-22T21:40:55.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.093+0000 7fe7b67fc640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 0x7fe79807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:55.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.093+0000 7fe7cd848640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 0x7fe79807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:55.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.093+0000 7fe7cd848640 10 monclient: get_auth_request con 0x7fe798078e50 auth_method 0 2026-02-22T21:40:55.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.094+0000 7fe7b67fc640 10 client.4928.objecter ms_handle_connect 0x7fe798078e50 2026-02-22T21:40:55.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.094+0000 7fe7cd848640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 0x7fe79807b310 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8009c40 tx=0x7fe7b8009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:55.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.189+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe7c405dc80 con 0x7fe7c815f3e0 2026-02-22T21:40:55.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.189+0000 7fe7b67fc640 10 client.4928.objecter ms_dispatch2 0x7fe7c8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:55.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.189+0000 7fe7b67fc640 3 client.4928.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:55.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.189+0000 7fe7b67fc640 3 client.4928.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:55.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.189+0000 7fe7b67fc640 20 client.4928.objecter dump_active .. 0 homeless 2026-02-22T21:40:55.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.195+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe7c4022020 con 0x7fe7c815f3e0 2026-02-22T21:40:55.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.195+0000 7fe7b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:55.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.195+0000 7fe7b67fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:55.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.296+0000 7fe7cfad3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:55.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.296+0000 7fe7cfad3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:55.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.296+0000 7fe7cfad3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:40:55.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.296+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe7c8159810 con 0x7fe7c815f3e0 2026-02-22T21:40:55.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.299+0000 7fe7b67fc640 1 -- 10.20.193.51:0/1461820829 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe7c4061c40 con 0x7fe7c815f3e0 2026-02-22T21:40:55.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.299+0000 7fe7b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:55.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.299+0000 7fe7b67fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:55.301 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:55.302 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:55.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 10 librados: watch_flush enter 2026-02-22T21:40:55.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 10 librados: watch_flush exit 2026-02-22T21:40:55.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 20 client.4928.objecter shutdown clearing up homeless session... 2026-02-22T21:40:55.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 10 client.4928.objecter successfully canceled tick 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 msgr2=0x7fe79807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 0x7fe79807b310 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8009c40 tx=0x7fe7b8009290 comp rx=0 tx=0).stop 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 10 monclient: shutdown 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 msgr2=0x7fe7c8158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400cb10 tx=0x7fe7c400a060 comp rx=0 tx=0).stop 2026-02-22T21:40:55.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 shutdown_connections 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe798078e50 0x7fe79807b310 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe7c815f3e0 0x7fe7c8158280 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe7c8152b00 0x7fe7c8074210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 --2- 10.20.193.51:0/1461820829 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe7c8152720 0x7fe7c8156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 >> 10.20.193.51:0/1461820829 conn(0x7fe7c80d1b50 msgr2=0x7fe7c8151690 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 shutdown_connections 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 -- 10.20.193.51:0/1461820829 wait complete. 2026-02-22T21:40:55.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:55.300+0000 7fe7cfad3640 1 librados: shutdown 2026-02-22T21:40:58.311 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:40:58.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: get_monmap_and_config 2026-02-22T21:40:58.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: build_initial_monmap 2026-02-22T21:40:58.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: monmap: 2026-02-22T21:40:58.383 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:58.383 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:58.383 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:40:58.377728+0000 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:40:58.377728+0000 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:40:58.384 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 1 Processor -- start 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 1 -- start start 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: init 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:58.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.377+0000 7fa5ceabd640 10 monclient: _add_conns ranks=[1,3,2,0,5,4] 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fa5c8150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa5c815c710 0x7fa5c80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c815c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5c8150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c815a920 con 0x7fa5c8150220 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c815af60 con 0x7fa5c8150600 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:58.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c815b240 con 0x7fa5c815c710 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5ceabd640 10 monclient(hunting): authenticate will time out at 1333.958074s 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa5c815c710 0x7fa5c80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5cc832640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fa5c8150220 0x7fa5c815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:45632/0 (socket says 10.20.193.51:45632) 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5cc832640 1 -- 10.20.193.51:0/3447767125 learned_addr learned my addr 10.20.193.51:0/3447767125 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): get_auth_request con 0x7fa5c815c710 auth_method 0 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:58.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.378+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 1 -- 10.20.193.51:0/3447767125 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392012398 0 0) 0x7fa5c815a920 con 0x7fa5c8150220 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:58.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 10 monclient(hunting): my global_id is 4957 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bf7fe640 1 -- 10.20.193.51:0/3447767125 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5b4003620 con 0x7fa5c8150220 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 10 monclient(hunting): handle_auth_done global_id 4962 payload 274 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 1 -- 10.20.193.51:0/3447767125 >> v1:10.20.193.125:6789/0 conn(0x7fa5c8150600 legacy=0x7fa5c806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 1 -- 10.20.193.51:0/3447767125 >> v1:10.20.193.68:6789/0 conn(0x7fa5c8150220 legacy=0x7fa5c815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 10 monclient: _finish_hunting 0 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 1 monclient: found mon.noname-a 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:58.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.379+0000 7fa5bffff640 1 -- 10.20.193.51:0/3447767125 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c815b520 con 0x7fa5c815c710 2026-02-22T21:40:58.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.380+0000 7fa5bffff640 1 --2- 10.20.193.51:0/3447767125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c815c710 0x7fa5c80c73a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098d0 tx=0x7fa5b002ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccd8a4c4132f510 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:58.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 1 -- 10.20.193.51:0/3447767125 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b003c070 con 0x7fa5c815c710 2026-02-22T21:40:58.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:58.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: dump: 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.402 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: _finish_auth 0 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 1 -- 10.20.193.51:0/3447767125 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b002fbf0 con 0x7fa5c815c710 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 5 monclient: authenticate success, global_id 4962 2026-02-22T21:40:58.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 1 -- 10.20.193.51:0/3447767125 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0037440 con 0x7fa5c815c710 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5bf7fe640 10 monclient: dump: 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:58.404 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 10 monclient: get_monmap_and_config success 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 4 set_mon_vals no callback set 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 10 monclient: shutdown 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 -- 10.20.193.51:0/3447767125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c815c710 msgr2=0x7fa5c80c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/3447767125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c815c710 0x7fa5c80c73a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00098d0 tx=0x7fa5b002ef20 comp rx=0 tx=0).stop 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 -- 10.20.193.51:0/3447767125 shutdown_connections 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/3447767125 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c815c710 0x7fa5c80c73a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 -- 10.20.193.51:0/3447767125 >> 10.20.193.51:0/3447767125 conn(0x7fa5c80cd7b0 msgr2=0x7fa5c80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 -- 10.20.193.51:0/3447767125 shutdown_connections 2026-02-22T21:40:58.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.382+0000 7fa5ceabd640 1 -- 10.20.193.51:0/3447767125 wait complete. 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient: build_initial_monmap 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient: monmap: 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:40:58.406 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 librados: starting msgr at 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 librados: starting objecter 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 Processor -- start 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 -- start start 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 librados: setting wanted keys 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 librados: calling monclient init 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient: init 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient: _reopen_session rank -1 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:58.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c8150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5c8150600 0x7fa5c8073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c8150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5c815c710 0x7fa5c814c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c815c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c80703b0 con 0x7fa5c8150220 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c815bae0 con 0x7fa5c8150600 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c8070530 con 0x7fa5c815c710 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): _renew_subs 2026-02-22T21:40:58.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.383+0000 7fa5ceabd640 10 monclient(hunting): authenticate will time out at 1333.963991s 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5c8150600 0x7fa5c8073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:34802/0 (socket says 10.20.193.51:34802) 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 1 -- 10.20.193.51:0/631845146 learned_addr learned my addr 10.20.193.51:0/631845146 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): get_auth_request con 0x7fa5c8150220 auth_method 0 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cd033640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5c815c710 0x7fa5c814c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): get_auth_request con 0x7fa5c8150600 auth_method 0 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:58.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cc832640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cd033640 10 monclient(hunting): get_auth_request con 0x7fa5c815c710 auth_method 0 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cd033640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cd033640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5cd033640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.384+0000 7fa5bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.385+0000 7fa5cd033640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:40:58.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.385+0000 7fa5cd033640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.385+0000 7fa5cd033640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 10 monclient(hunting): handle_auth_done global_id 4965 payload 824 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 -- 10.20.193.51:0/631845146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5c815c710 msgr2=0x7fa5c814c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5c815c710 0x7fa5c814c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 -- 10.20.193.51:0/631845146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5c8150600 msgr2=0x7fa5c8073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5c8150600 0x7fa5c8073ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 10 monclient: _finish_hunting 0 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 monclient: found mon.noname-a 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.386+0000 7fa5cc832640 1 -- 10.20.193.51:0/631845146 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c814d850 con 0x7fa5c8150220 2026-02-22T21:40:58.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.387+0000 7fa5cc832640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c4b0 tx=0x7fa5b800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.387+0000 7fa5bd7fa640 10 client.?.objecter ms_handle_connect 0x7fa5c8150220 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.387+0000 7fa5bd7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8017070 con 0x7fa5c8150220 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: dump: 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.412 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: _finish_auth 0 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b8005de0 con 0x7fa5c8150220 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8007270 con 0x7fa5c8150220 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 5 monclient: authenticate success, global_id 4965 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 10 monclient: _renew_subs 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5beffd640 4 set_mon_vals no callback set 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c80c2d30 con 0x7fa5c8150220 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5bd7fa640 10 monclient: dump: 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:40:58.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 10 client.4965.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 10 monclient: _renew_subs 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c80c32f0 con 0x7fa5c8150220 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.388+0000 7fa5ceabd640 1 librados: init done 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.389+0000 7fa5ceabd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.389+0000 7fa5ceabd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.389+0000 7fa5ceabd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.389+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c80c73a0 con 0x7fa5c8150220 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.497+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa5b8007410 con 0x7fa5c8150220 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.498+0000 7fa5bd7fa640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 0x7fa59807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.498+0000 7fa5bffff640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 0x7fa59807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.498+0000 7fa5bffff640 10 monclient: get_auth_request con 0x7fa598078e00 auth_method 0 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.498+0000 7fa5bffff640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 0x7fa59807b2c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0039000 tx=0x7fa5b0033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:40:58.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.498+0000 7fa5bd7fa640 10 client.4965.objecter ms_handle_connect 0x7fa598078e00 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa5b8007730 con 0x7fa5c8150220 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 10 client.4965.objecter ms_dispatch2 0x7fa5c8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 3 client.4965.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 3 client.4965.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 20 client.4965.objecter dump_active .. 0 homeless 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa5b80613b0 con 0x7fa5c8150220 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:40:58.508 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.506+0000 7fa5bd7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:40:58.608 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.606+0000 7fa5ceabd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:40:58.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.606+0000 7fa5ceabd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:58.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.606+0000 7fa5ceabd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:40:58.609 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.606+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa5c8074da0 con 0x7fa5c8150220 2026-02-22T21:40:58.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5bd7fa640 1 -- 10.20.193.51:0/631845146 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa5b80656b0 con 0x7fa5c8150220 2026-02-22T21:40:58.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:40:58.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5bd7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:40:58.612 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:40:58.612 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:40:58.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5ceabd640 10 librados: watch_flush enter 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5ceabd640 10 librados: watch_flush exit 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5ceabd640 20 client.4965.objecter shutdown clearing up homeless session... 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.610+0000 7fa5ceabd640 10 client.4965.objecter successfully canceled tick 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 msgr2=0x7fa59807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 0x7fa59807b2c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0039000 tx=0x7fa5b0033000 comp rx=0 tx=0).stop 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 10 monclient: shutdown 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:40:58.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 msgr2=0x7fa5c814d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c4b0 tx=0x7fa5b800c980 comp rx=0 tx=0).stop 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 shutdown_connections 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa598078e00 0x7fa59807b2c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa5c815c710 0x7fa5c814c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa5c8150600 0x7fa5c8073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 --2- 10.20.193.51:0/631845146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa5c8150220 0x7fa5c814d310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 >> 10.20.193.51:0/631845146 conn(0x7fa5c80cd7b0 msgr2=0x7fa5c80cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 shutdown_connections 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 -- 10.20.193.51:0/631845146 wait complete. 2026-02-22T21:40:58.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:40:58.611+0000 7fa5ceabd640 1 librados: shutdown 2026-02-22T21:41:01.521 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:32.674 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.546 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:32.700 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.547 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:32.698 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.548 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:32.700 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.555 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:32.709 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.556 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:32.708 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.572 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:32.688 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.583 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:32.699 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.592 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:32.708 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:01.622 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.690+0000 7f80fee61640 10 monclient: get_monmap_and_config 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.690+0000 7f80fee61640 10 monclient: build_initial_monmap 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.690+0000 7f80fee61640 10 monclient: monmap: 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:01.691374+0000 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:01.691374+0000 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:01.692 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:01.693 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 Processor -- start 2026-02-22T21:41:01.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 -- start start 2026-02-22T21:41:01.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient: init 2026-02-22T21:41:01.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient: _add_conns ranks=[1,0,3,4,2,5] 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f80f8150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80f815c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f80f80703b0 0x7f80f80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-a con 0x7f80f80703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f815a920 con 0x7f80f815c710 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f815af60 con 0x7f80f8150220 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f80f815b240 con 0x7f80f80703b0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.691+0000 7f80fee61640 10 monclient(hunting): authenticate will time out at 1337.271763s 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.692+0000 7f80fcbd6640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f80f8150220 0x7f80f815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:45640/0 (socket says 10.20.193.51:45640) 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.692+0000 7f80fcbd6640 1 -- 10.20.193.51:0/3095542675 learned_addr learned my addr 10.20.193.51:0/3095542675 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 1 --2- 10.20.193.51:0/3095542675 >> v2:10.20.193.51:3300/0 conn(0x7f80f80703b0 0x7f80f80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): get_auth_request con 0x7f80f80703b0 auth_method 0 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:01.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:01.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 1 -- 10.20.193.51:0/3095542675 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423055498 0 0) 0x7f80f815af60 con 0x7f80f8150220 2026-02-22T21:41:01.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 10 monclient(hunting): my global_id is 4963 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80ef7fe640 1 -- 10.20.193.51:0/3095542675 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80e4003620 con 0x7f80f8150220 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.693+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_done global_id 4968 payload 274 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3095542675 >> v1:10.20.193.68:6789/0 conn(0x7f80f8150220 legacy=0x7f80f815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3095542675 >> v1:10.20.193.51:6789/0 conn(0x7f80f815c710 legacy=0x7f80f806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 10 monclient: _finish_hunting 0 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 1 monclient: found mon.noname-a 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.695+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3095542675 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f815b520 con 0x7f80f80703b0 2026-02-22T21:41:01.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.697+0000 7f80fd3d7640 1 --2- 10.20.193.51:0/3095542675 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f80703b0 0x7f80f80c73a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009dc0 tx=0x7f80f402f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa9e4913716d1e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 1 -- 10.20.193.51:0/3095542675 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f403c070 con 0x7f80f80703b0 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 10 monclient: dump: 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:01.700 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80fee61640 5 monclient: authenticate success, global_id 4968 2026-02-22T21:41:01.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.698+0000 7f80fee61640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 1 -- 10.20.193.51:0/3095542675 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f4037440 con 0x7f80f80703b0 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 1 -- 10.20.193.51:0/3095542675 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f40377f0 con 0x7f80f80703b0 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80ef7fe640 10 monclient: dump: 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:01.704 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 4 set_mon_vals no callback set 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 10 monclient: shutdown 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 -- 10.20.193.51:0/3095542675 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f80703b0 msgr2=0x7f80f80c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 --2- 10.20.193.51:0/3095542675 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f80703b0 0x7f80f80c73a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009dc0 tx=0x7f80f402f2f0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 -- 10.20.193.51:0/3095542675 shutdown_connections 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 --2- 10.20.193.51:0/3095542675 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f80703b0 0x7f80f80c73a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 -- 10.20.193.51:0/3095542675 >> 10.20.193.51:0/3095542675 conn(0x7f80f80cd7b0 msgr2=0x7f80f80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 -- 10.20.193.51:0/3095542675 shutdown_connections 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.699+0000 7f80fee61640 1 -- 10.20.193.51:0/3095542675 wait complete. 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 10 monclient: build_initial_monmap 2026-02-22T21:41:01.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 10 monclient: monmap: 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 librados: starting msgr at 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 librados: starting objecter 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 Processor -- start 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 -- start start 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 librados: setting wanted keys 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 1 librados: calling monclient init 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.700+0000 7f80fee61640 10 monclient: init 2026-02-22T21:41:01.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f80f80703b0 0x7f80f80c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-c con 0x7f80f80703b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f80f8150220 0x7f80f81f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-b con 0x7f80f8150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): picked mon.noname-a con 0x7f80f815c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f80f8150600 con 0x7f80f815c710 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f80f815bae0 con 0x7f80f8150220 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f80f8150780 con 0x7f80f80703b0 2026-02-22T21:41:01.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fee61640 10 monclient(hunting): authenticate will time out at 1337.281203s 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80effff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f80f8150220 0x7f80f81f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fcbd6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f80f80703b0 0x7f80f80c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:34824/0 (socket says 10.20.193.51:34824) 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3818694283 learned_addr learned my addr 10.20.193.51:0/3818694283 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): get_auth_request con 0x7f80f815c710 auth_method 0 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80effff640 10 monclient(hunting): get_auth_request con 0x7f80f8150220 auth_method 0 2026-02-22T21:41:01.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80effff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80effff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fcbd6640 10 monclient(hunting): get_auth_request con 0x7f80f80703b0 auth_method 0 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fcbd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fcbd6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fcbd6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.701+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:01.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:01.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80fcbd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:01.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80fcbd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:01.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.702+0000 7f80fcbd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:01.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 10 monclient(hunting): handle_auth_done global_id 4974 payload 824 2026-02-22T21:41:01.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3818694283 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f80f80703b0 msgr2=0x7f80f80c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f80f80703b0 0x7f80f80c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3818694283 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f80f8150220 msgr2=0x7f80f81f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f80f8150220 0x7f80f81f4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 10 monclient: _finish_hunting 0 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 monclient: found mon.noname-a 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.703+0000 7f80fd3d7640 1 -- 10.20.193.51:0/3818694283 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f814d440 con 0x7f80f815c710 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.704+0000 7f80fd3d7640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f80f403a040 tx=0x7f80f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.704+0000 7f80ed7fa640 10 client.?.objecter ms_handle_connect 0x7f80f815c710 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.704+0000 7f80ed7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f403c050 con 0x7f80f815c710 2026-02-22T21:41:01.713 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: dump: 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.714 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f4048070 con 0x7f80f815c710 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4004040 con 0x7f80f815c710 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80ed7fa640 10 monclient: dump: 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.715 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80eeffd640 4 set_mon_vals no callback set 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 5 monclient: authenticate success, global_id 4974 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _renew_subs 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f80c31f0 con 0x7f80f815c710 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 client.4974.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:01.716 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _renew_subs 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f80c37b0 con 0x7f80f815c710 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 1 librados: init done 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.717 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.706+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f8074da0 con 0x7f80f815c710 2026-02-22T21:41:01.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f80f40041e0 con 0x7f80f815c710 2026-02-22T21:41:01.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80ed7fa640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 0x7f80c807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:01.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80fcbd6640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 0x7f80c807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:01.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80fcbd6640 10 monclient: get_auth_request con 0x7f80c8078e80 auth_method 0 2026-02-22T21:41:01.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80fcbd6640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 0x7f80c807b340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80e8007b60 tx=0x7f80e8007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:01.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.718+0000 7f80ed7fa640 10 client.4974.objecter ms_handle_connect 0x7f80c8078e80 2026-02-22T21:41:01.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.721+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f80f4084130 con 0x7f80f815c710 2026-02-22T21:41:01.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.721+0000 7f80ed7fa640 10 client.4974.objecter ms_dispatch2 0x7f80f8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:01.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.721+0000 7f80ed7fa640 3 client.4974.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:01.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.721+0000 7f80ed7fa640 3 client.4974.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:01.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.721+0000 7f80ed7fa640 20 client.4974.objecter dump_active .. 0 homeless 2026-02-22T21:41:01.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.822+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f80f40880f0 con 0x7f80f815c710 2026-02-22T21:41:01.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.822+0000 7f80ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:01.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.822+0000 7f80ed7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:01.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.924+0000 7f80fee61640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:01.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.924+0000 7f80fee61640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:01.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.924+0000 7f80fee61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:01.927 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.925+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f80f814f550 con 0x7f80f815c710 2026-02-22T21:41:01.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.927+0000 7f80ed7fa640 1 -- 10.20.193.51:0/3818694283 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f80f408c3f0 con 0x7f80f815c710 2026-02-22T21:41:01.928 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.927+0000 7f80ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:01.929 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.927+0000 7f80ed7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:01.929 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:01.929 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:01.930 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 10 librados: watch_flush enter 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 10 librados: watch_flush exit 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 20 client.4974.objecter shutdown clearing up homeless session... 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 10 client.4974.objecter successfully canceled tick 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 msgr2=0x7f80c807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 0x7f80c807b340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80e8007b60 tx=0x7f80e8007ab0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 10 monclient: shutdown 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 msgr2=0x7f80f80c2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f80f403a040 tx=0x7f80f4004b40 comp rx=0 tx=0).stop 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 shutdown_connections 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f80c8078e80 0x7f80c807b340 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.931 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f80f815c710 0x7f80f80c2a70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f80f8150220 0x7f80f81f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 --2- 10.20.193.51:0/3818694283 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f80f80703b0 0x7f80f80c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 >> 10.20.193.51:0/3818694283 conn(0x7f80f80cd7b0 msgr2=0x7f80f80c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 shutdown_connections 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 -- 10.20.193.51:0/3818694283 wait complete. 2026-02-22T21:41:01.932 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:01.928+0000 7f80fee61640 1 librados: shutdown 2026-02-22T21:41:04.939 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:05.023 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: get_monmap_and_config 2026-02-22T21:41:05.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: build_initial_monmap 2026-02-22T21:41:05.024 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: monmap: 2026-02-22T21:41:05.024 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:05.024 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:05.024 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:05.004728+0000 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:05.004728+0000 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:05.025 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:05.026 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:05.026 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:05.026 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:05.026 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.026 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 1 Processor -- start 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 1 -- start start 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: init 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 10 monclient: _add_conns ranks=[4,3,2,5,0,1] 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.004+0000 7f64f312c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f64ec150220 0x7f64ec0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-b con 0x7f64ec150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:05.027 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 0x7f64ec15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-a con 0x7f64ec150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64ec15c540 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64ec15c9b0 con 0x7f64ec15c540 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f64ec15a920 con 0x7f64ec150600 2026-02-22T21:41:05.028 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f64ec15af60 con 0x7f64ec150220 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f312c640 10 monclient(hunting): authenticate will time out at 1340.585035s 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f0ea1640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f64ec150220 0x7f64ec0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f16a2640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f64ec15c540 0x7f64ec06fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:50678/0 (socket says 10.20.193.51:50678) 2026-02-22T21:41:05.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f16a2640 1 -- 10.20.193.51:0/1044740181 learned_addr learned my addr 10.20.193.51:0/1044740181 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64e3fff640 1 --2- 10.20.193.51:0/1044740181 >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 0x7f64ec15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f0ea1640 10 monclient(hunting): get_auth_request con 0x7f64ec150220 auth_method 0 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f0ea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f0ea1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64f0ea1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64e3fff640 10 monclient(hunting): get_auth_request con 0x7f64ec150600 auth_method 0 2026-02-22T21:41:05.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64e3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.005+0000 7f64e3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:05.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 1 -- 10.20.193.51:0/1044740181 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807832048 0 0) 0x7f64ec15c9b0 con 0x7f64ec15c540 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 10 monclient(hunting): _negotiate 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 10 monclient(hunting): my global_id is 4940 2026-02-22T21:41:05.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.006+0000 7f64e37fe640 1 -- 10.20.193.51:0/1044740181 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64d8003620 con 0x7f64ec15c540 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_done global_id 4969 payload 274 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 1 -- 10.20.193.51:0/1044740181 >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 msgr2=0x7f64ec15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 1 --2- 10.20.193.51:0/1044740181 >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 0x7f64ec15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 1 -- 10.20.193.51:0/1044740181 >> v1:10.20.193.125:6789/0 conn(0x7f64ec15c540 legacy=0x7f64ec06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 10 monclient: _finish_hunting 0 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 1 monclient: found mon.noname-b 2026-02-22T21:41:05.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64f0ea1640 1 -- 10.20.193.51:0/1044740181 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec15b240 con 0x7f64ec150220 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.007+0000 7f64e3fff640 1 --2- 10.20.193.51:0/1044740181 >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 0x7f64ec15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.008+0000 7f64f0ea1640 1 --2- 10.20.193.51:0/1044740181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec150220 0x7f64ec0c73a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f64ec070940 tx=0x7f64dc02eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8213b07dad34470 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 1 -- 10.20.193.51:0/1044740181 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc03c070 con 0x7f64ec150220 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:05.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 10 monclient: dump: 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:05.035 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64f312c640 5 monclient: authenticate success, global_id 4969 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.009+0000 7f64f312c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:05.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 1 -- 10.20.193.51:0/1044740181 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc02fb80 con 0x7f64ec150220 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 1 -- 10.20.193.51:0/1044740181 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc037440 con 0x7f64ec150220 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64e37fe640 10 monclient: dump: 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:05.037 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 4 set_mon_vals no callback set 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 10 monclient: shutdown 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 -- 10.20.193.51:0/1044740181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec150220 msgr2=0x7f64ec0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 --2- 10.20.193.51:0/1044740181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec150220 0x7f64ec0c73a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f64ec070940 tx=0x7f64dc02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 -- 10.20.193.51:0/1044740181 shutdown_connections 2026-02-22T21:41:05.039 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 --2- 10.20.193.51:0/1044740181 >> v2:10.20.193.51:3300/0 conn(0x7f64ec150600 0x7f64ec15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 --2- 10.20.193.51:0/1044740181 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec150220 0x7f64ec0c73a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 -- 10.20.193.51:0/1044740181 >> 10.20.193.51:0/1044740181 conn(0x7f64ec0cd7b0 msgr2=0x7f64ec0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 -- 10.20.193.51:0/1044740181 shutdown_connections 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 1 -- 10.20.193.51:0/1044740181 wait complete. 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 10 monclient: build_initial_monmap 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.010+0000 7f64f312c640 10 monclient: monmap: 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:05.040 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 librados: starting msgr at 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 librados: starting objecter 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 Processor -- start 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 -- start start 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 librados: setting wanted keys 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 librados: calling monclient init 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient: init 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:05.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f64ec150220 0x7f64ec073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-c con 0x7f64ec150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f64ec150600 0x7f64ec156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-a con 0x7f64ec150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): picked mon.noname-b con 0x7f64ec15c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec15bae0 con 0x7f64ec150600 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:05.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec15b800 con 0x7f64ec15c540 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec0703b0 con 0x7f64ec150220 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f312c640 10 monclient(hunting): authenticate will time out at 1340.591602s 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.011+0000 7f64f16a2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f64ec150220 0x7f64ec073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f64ec150600 0x7f64ec156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:38142/0 (socket says 10.20.193.51:38142) 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 1 -- 10.20.193.51:0/3309208744 learned_addr learned my addr 10.20.193.51:0/3309208744 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:05.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): get_auth_request con 0x7f64ec15c540 auth_method 0 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): get_auth_request con 0x7f64ec150220 auth_method 0 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): get_auth_request con 0x7f64ec150600 auth_method 0 2026-02-22T21:41:05.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f16a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64f0ea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:05.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.012+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 10 monclient(hunting): handle_auth_done global_id 4972 payload 824 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 -- 10.20.193.51:0/3309208744 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f64ec150220 msgr2=0x7f64ec073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f64ec150220 0x7f64ec073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 -- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f64ec150600 msgr2=0x7f64ec156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f64ec150600 0x7f64ec156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 10 monclient: _finish_hunting 0 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 monclient: found mon.noname-b 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:05.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.013+0000 7f64f16a2640 1 -- 10.20.193.51:0/3309208744 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec074380 con 0x7f64ec15c540 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.014+0000 7f64f16a2640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64e800caa0 tx=0x7f64e800cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.014+0000 7f64e17fa640 10 client.?.objecter ms_handle_connect 0x7f64ec15c540 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.014+0000 7f64e17fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e800ac40 con 0x7f64ec15c540 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 10 monclient: dump: 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:05.048 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:05.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 5 monclient: authenticate success, global_id 4972 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 10 monclient: _renew_subs 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec155e50 con 0x7f64ec15c540 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 10 client.4972.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 10 monclient: _renew_subs 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec156410 con 0x7f64ec15c540 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.015+0000 7f64f312c640 1 librados: init done 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64f312c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:05.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64f312c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64f312c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec15c000 con 0x7f64ec15c540 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e800ade0 con 0x7f64ec15c540 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e8003370 con 0x7f64ec15c540 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e17fa640 10 monclient: dump: 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:05.051 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:05.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.016+0000 7f64e2ffd640 4 set_mon_vals no callback set 2026-02-22T21:41:05.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.123+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f64e80256f0 con 0x7f64ec15c540 2026-02-22T21:41:05.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.124+0000 7f64e17fa640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 0x7f64bc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:05.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.124+0000 7f64f0ea1640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 0x7f64bc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:05.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.124+0000 7f64f0ea1640 10 monclient: get_auth_request con 0x7f64bc078e00 auth_method 0 2026-02-22T21:41:05.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.124+0000 7f64f0ea1640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 0x7f64bc07b2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64dc039000 tx=0x7f64dc033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:05.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.124+0000 7f64e17fa640 10 client.4972.objecter ms_handle_connect 0x7f64bc078e00 2026-02-22T21:41:05.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.126+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f64e805e8a0 con 0x7f64ec15c540 2026-02-22T21:41:05.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.126+0000 7f64e17fa640 10 client.4972.objecter ms_dispatch2 0x7f64ec000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:05.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.126+0000 7f64e17fa640 3 client.4972.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:05.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.126+0000 7f64e17fa640 3 client.4972.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:05.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.126+0000 7f64e17fa640 20 client.4972.objecter dump_active .. 0 homeless 2026-02-22T21:41:05.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.132+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f64e8062860 con 0x7f64ec15c540 2026-02-22T21:41:05.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.132+0000 7f64e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:05.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.132+0000 7f64e17fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:05.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.231+0000 7f64f312c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:05.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.231+0000 7f64f312c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:05.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.231+0000 7f64f312c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:05.233 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.231+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f64ec157d50 con 0x7f64ec15c540 2026-02-22T21:41:05.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.233+0000 7f64e17fa640 1 -- 10.20.193.51:0/3309208744 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f64e8066b60 con 0x7f64ec15c540 2026-02-22T21:41:05.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.233+0000 7f64e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:05.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.233+0000 7f64e17fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:05.235 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:05.236 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:05.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 10 librados: watch_flush enter 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 10 librados: watch_flush exit 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 20 client.4972.objecter shutdown clearing up homeless session... 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 10 client.4972.objecter successfully canceled tick 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 msgr2=0x7f64bc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 0x7f64bc07b2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64dc039000 tx=0x7f64dc033000 comp rx=0 tx=0).stop 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 10 monclient: shutdown 2026-02-22T21:41:05.238 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 msgr2=0x7f64ec1f7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64e800caa0 tx=0x7f64e800cf70 comp rx=0 tx=0).stop 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 shutdown_connections 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f64bc078e00 0x7f64bc07b2c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f64ec15c540 0x7f64ec1f7120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f64ec150600 0x7f64ec156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 --2- 10.20.193.51:0/3309208744 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f64ec150220 0x7f64ec073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:05.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 >> 10.20.193.51:0/3309208744 conn(0x7f64ec0cd7b0 msgr2=0x7f64ec06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:05.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 shutdown_connections 2026-02-22T21:41:05.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 -- 10.20.193.51:0/3309208744 wait complete. 2026-02-22T21:41:05.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:05.234+0000 7f64f312c640 1 librados: shutdown 2026-02-22T21:41:08.245 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:08.315 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.313+0000 7f1951d44640 10 monclient: get_monmap_and_config 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.313+0000 7f1951d44640 10 monclient: build_initial_monmap 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.313+0000 7f1951d44640 10 monclient: monmap: 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:08.314336+0000 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:08.314336+0000 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:08.316 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 Processor -- start 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 -- start start 2026-02-22T21:41:08.317 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient: init 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient: _add_conns ranks=[3,4,5,1,0,2] 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f194c150220 0x7f194c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-a con 0x7f194c150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f194c150600 0x7f194c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-b con 0x7f194c150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:08.325 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f194c15c540 0x7f194c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-c con 0x7f194c15c540 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f194c15c9b0 con 0x7f194c150220 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f194c15a920 con 0x7f194c150600 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f194c15af60 con 0x7f194c15c540 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f1951d44640 10 monclient(hunting): authenticate will time out at 1343.894645s 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.314+0000 7f194b7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f194c150220 0x7f194c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f194c150220 0x7f194c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:56844/0 (socket says 10.20.193.51:56844) 2026-02-22T21:41:08.326 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 1 -- 10.20.193.51:0/1203232812 learned_addr learned my addr 10.20.193.51:0/1203232812 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): get_auth_request con 0x7f194c150220 auth_method 0 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.68:3300/0 conn(0x7f194c150600 0x7f194c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.125:3300/0 conn(0x7f194c15c540 0x7f194c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): get_auth_request con 0x7f194c150600 auth_method 0 2026-02-22T21:41:08.327 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): get_auth_request con 0x7f194c15c540 auth_method 0 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.315+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 10 monclient(hunting): handle_auth_done global_id 4983 payload 274 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 -- 10.20.193.51:0/1203232812 >> v2:10.20.193.125:3300/0 conn(0x7f194c15c540 msgr2=0x7f194c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.125:3300/0 conn(0x7f194c15c540 0x7f194c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 -- 10.20.193.51:0/1203232812 >> v2:10.20.193.68:3300/0 conn(0x7f194c150600 msgr2=0x7f194c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.68:3300/0 conn(0x7f194c150600 0x7f194c15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.316+0000 7f194b7fe640 1 -- 10.20.193.51:0/1203232812 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c15b240 con 0x7f194c150220 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.317+0000 7f194b7fe640 1 --2- 10.20.193.51:0/1203232812 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150220 0x7f194c0c73a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f194c074940 tx=0x7f193802ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df63b94d9aa9f02e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 1 -- 10.20.193.51:0/1203232812 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193803c070 con 0x7f194c150220 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:08.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: dump: 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:08.330 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 5 monclient: authenticate success, global_id 4983 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 1 -- 10.20.193.51:0/1203232812 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193802fbd0 con 0x7f194c150220 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 1 -- 10.20.193.51:0/1203232812 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938037440 con 0x7f194c150220 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 4 set_mon_vals no callback set 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:08.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f194a7fc640 10 monclient: dump: 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 10 monclient: shutdown 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 -- 10.20.193.51:0/1203232812 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150220 msgr2=0x7f194c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 --2- 10.20.193.51:0/1203232812 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150220 0x7f194c0c73a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f194c074940 tx=0x7f193802ef50 comp rx=0 tx=0).stop 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 -- 10.20.193.51:0/1203232812 shutdown_connections 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.125:3300/0 conn(0x7f194c15c540 0x7f194c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 --2- 10.20.193.51:0/1203232812 >> v2:10.20.193.68:3300/0 conn(0x7f194c150600 0x7f194c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 --2- 10.20.193.51:0/1203232812 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150220 0x7f194c0c73a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 -- 10.20.193.51:0/1203232812 >> 10.20.193.51:0/1203232812 conn(0x7f194c0cd7b0 msgr2=0x7f194c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 -- 10.20.193.51:0/1203232812 shutdown_connections 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 -- 10.20.193.51:0/1203232812 wait complete. 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 10 monclient: build_initial_monmap 2026-02-22T21:41:08.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 10 monclient: monmap: 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 librados: starting msgr at 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.319+0000 7f1951d44640 1 librados: starting objecter 2026-02-22T21:41:08.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 Processor -- start 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 -- start start 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 librados: setting wanted keys 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 librados: calling monclient init 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient: init 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f194c150220 0x7f194c1f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-c con 0x7f194c150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-a con 0x7f194c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f194c15c540 0x7f194c14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): picked mon.noname-b con 0x7f194c15c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f194c15bae0 con 0x7f194c150600 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f194c15b800 con 0x7f194c15c540 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f194c0701e0 con 0x7f194c150220 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f1951d44640 10 monclient(hunting): authenticate will time out at 1343.900494s 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:56846/0 (socket says 10.20.193.51:56846) 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 1 -- 10.20.193.51:0/114383384 learned_addr learned my addr 10.20.193.51:0/114383384 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:08.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 10 monclient(hunting): get_auth_request con 0x7f194c150600 auth_method 0 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194bfff640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f194c15c540 0x7f194c14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.320+0000 7f194b7fe640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f194c150220 0x7f194c1f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): get_auth_request con 0x7f194c15c540 auth_method 0 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): get_auth_request con 0x7f194c150220 auth_method 0 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.321+0000 7f194bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 10 monclient(hunting): handle_auth_done global_id 4986 payload 824 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 -- 10.20.193.51:0/114383384 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f194c150220 msgr2=0x7f194c1f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:08.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f194c150220 0x7f194c1f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 -- 10.20.193.51:0/114383384 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f194c15c540 msgr2=0x7f194c14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f194c15c540 0x7f194c14c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 monclient: found mon.noname-a 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.322+0000 7f194affd640 1 -- 10.20.193.51:0/114383384 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c0c1e40 con 0x7f194c150600 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.323+0000 7f194affd640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f193c00ebe0 tx=0x7f193c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.323+0000 7f1927fff640 10 client.?.objecter ms_handle_connect 0x7f194c150600 2026-02-22T21:41:08.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.323+0000 7f1927fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c017070 con 0x7f194c150600 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: dump: 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:08.342 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: _finish_auth 0 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193c00cc40 con 0x7f194c150600 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c013470 con 0x7f194c150600 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 5 monclient: authenticate success, global_id 4986 2026-02-22T21:41:08.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 10 monclient: _renew_subs 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c157000 con 0x7f194c150600 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1949ffb640 4 set_mon_vals no callback set 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1927fff640 10 monclient: dump: 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:08.344 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 10 client.4986.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 10 monclient: _renew_subs 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c1f18a0 con 0x7f194c150600 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.324+0000 7f1951d44640 1 librados: init done 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.325+0000 7f1951d44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.325+0000 7f1951d44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:08.345 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.325+0000 7f1951d44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.325+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c070570 con 0x7f194c150600 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.336+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f193c024060 con 0x7f194c150600 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.337+0000 7f1927fff640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 0x7f191c07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.337+0000 7f194b7fe640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 0x7f191c07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.337+0000 7f194b7fe640 10 monclient: get_auth_request con 0x7f191c078e30 auth_method 0 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.337+0000 7f194b7fe640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 0x7f191c07b2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f194c070940 tx=0x7f1938039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.337+0000 7f1927fff640 10 client.4986.objecter ms_handle_connect 0x7f191c078e30 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.338+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f193c05e260 con 0x7f194c150600 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.338+0000 7f1927fff640 10 client.4986.objecter ms_dispatch2 0x7f194c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.338+0000 7f1927fff640 3 client.4986.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.338+0000 7f1927fff640 3 client.4986.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:08.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.338+0000 7f1927fff640 20 client.4986.objecter dump_active .. 0 homeless 2026-02-22T21:41:08.440 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.439+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f193c062220 con 0x7f194c150600 2026-02-22T21:41:08.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.439+0000 7f1927fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:08.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.439+0000 7f1927fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:08.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.538+0000 7f1951d44640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:08.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.538+0000 7f1951d44640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:08.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.538+0000 7f1951d44640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:08.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.538+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f194c1568a0 con 0x7f194c150600 2026-02-22T21:41:08.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.542+0000 7f1927fff640 1 -- 10.20.193.51:0/114383384 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f193c066520 con 0x7f194c150600 2026-02-22T21:41:08.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.542+0000 7f1927fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:08.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.542+0000 7f1927fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:08.544 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:08.544 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:08.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 10 librados: watch_flush enter 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 10 librados: watch_flush exit 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 20 client.4986.objecter shutdown clearing up homeless session... 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 10 client.4986.objecter successfully canceled tick 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 msgr2=0x7f191c07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 0x7f191c07b2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f194c070940 tx=0x7f1938039040 comp rx=0 tx=0).stop 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 10 monclient: shutdown 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 msgr2=0x7f194c156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:08.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f193c00ebe0 tx=0x7f193c00c520 comp rx=0 tx=0).stop 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 shutdown_connections 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f191c078e30 0x7f191c07b2f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f194c15c540 0x7f194c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f194c150600 0x7f194c156ac0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 --2- 10.20.193.51:0/114383384 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f194c150220 0x7f194c1f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 >> 10.20.193.51:0/114383384 conn(0x7f194c0cd7b0 msgr2=0x7f194c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 shutdown_connections 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 -- 10.20.193.51:0/114383384 wait complete. 2026-02-22T21:41:08.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:08.543+0000 7f1951d44640 1 librados: shutdown 2026-02-22T21:41:11.554 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.619+0000 7f9d9326f640 10 monclient: get_monmap_and_config 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.619+0000 7f9d9326f640 10 monclient: build_initial_monmap 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.619+0000 7f9d9326f640 10 monclient: monmap: 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:11.620624+0000 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:11.620624+0000 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:11.621 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 Processor -- start 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 -- start start 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient: init 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient: _add_conns ranks=[0,4,1,3,2,5] 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d8c150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9d8c15c710 0x7f9d8c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-b con 0x7f9d8c15c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f9d8c150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c15a920 con 0x7f9d8c150220 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c15af60 con 0x7f9d8c150600 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:11.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c15b240 con 0x7f9d8c15c710 2026-02-22T21:41:11.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:11.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.620+0000 7f9d9326f640 10 monclient(hunting): authenticate will time out at 1347.200923s 2026-02-22T21:41:11.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d83fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9d8c15c710 0x7f9d8c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d90fe4640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f9d8c150220 0x7f9d8c15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:36170/0 (socket says 10.20.193.51:36170) 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d90fe4640 1 -- 10.20.193.51:0/14526445 learned_addr learned my addr 10.20.193.51:0/14526445 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d83fff640 10 monclient(hunting): get_auth_request con 0x7f9d8c15c710 auth_method 0 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d83fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.621+0000 7f9d83fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 1 -- 10.20.193.51:0/14526445 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3437649010 0 0) 0x7f9d8c15a920 con 0x7f9d8c150220 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 10 monclient(hunting): _negotiate 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 10 monclient(hunting): my global_id is 4989 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.622+0000 7f9d837fe640 1 -- 10.20.193.51:0/14526445 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d78003620 con 0x7f9d8c150220 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 10 monclient(hunting): handle_auth_done global_id 4981 payload 274 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 1 -- 10.20.193.51:0/14526445 >> v1:10.20.193.68:6789/0 conn(0x7f9d8c150600 legacy=0x7f9d8c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 1 -- 10.20.193.51:0/14526445 >> v1:10.20.193.51:6789/0 conn(0x7f9d8c150220 legacy=0x7f9d8c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 1 monclient: found mon.noname-b 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.623+0000 7f9d83fff640 1 -- 10.20.193.51:0/14526445 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c15b520 con 0x7f9d8c15c710 2026-02-22T21:41:11.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.625+0000 7f9d83fff640 1 --2- 10.20.193.51:0/14526445 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c15c710 0x7f9d8c0c73a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9d740098d0 tx=0x7f9d7402ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=899af82813ca8c41 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 1 -- 10.20.193.51:0/14526445 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7403c070 con 0x7f9d8c15c710 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: dump: 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:11.629 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 1 -- 10.20.193.51:0/14526445 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7402fbf0 con 0x7f9d8c15c710 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 5 monclient: authenticate success, global_id 4981 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 1 -- 10.20.193.51:0/14526445 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74037440 con 0x7f9d8c15c710 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d837fe640 10 monclient: dump: 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:11.630 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 4 set_mon_vals no callback set 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 10 monclient: shutdown 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 -- 10.20.193.51:0/14526445 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c15c710 msgr2=0x7f9d8c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 --2- 10.20.193.51:0/14526445 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c15c710 0x7f9d8c0c73a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9d740098d0 tx=0x7f9d7402ef20 comp rx=0 tx=0).stop 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 -- 10.20.193.51:0/14526445 shutdown_connections 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 --2- 10.20.193.51:0/14526445 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c15c710 0x7f9d8c0c73a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 -- 10.20.193.51:0/14526445 >> 10.20.193.51:0/14526445 conn(0x7f9d8c0cd7b0 msgr2=0x7f9d8c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 -- 10.20.193.51:0/14526445 shutdown_connections 2026-02-22T21:41:11.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.627+0000 7f9d9326f640 1 -- 10.20.193.51:0/14526445 wait complete. 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient: build_initial_monmap 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient: monmap: 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:11.633 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 librados: starting msgr at 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 librados: starting objecter 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 Processor -- start 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 -- start start 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 librados: setting wanted keys 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 librados: calling monclient init 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient: init 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-b con 0x7f9d8c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d8c150600 0x7f9d8c0c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-c con 0x7f9d8c150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8c15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0703b0 con 0x7f9d8c15c710 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c15bae0 con 0x7f9d8c150220 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c070530 con 0x7f9d8c150600 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.628+0000 7f9d9326f640 10 monclient(hunting): authenticate will time out at 1347.208883s 2026-02-22T21:41:11.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d83fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d8c150600 0x7f9d8c0c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:46584/0 (socket says 10.20.193.51:46584) 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:56852/0 (socket says 10.20.193.51:56852) 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 1 -- 10.20.193.51:0/712329438 learned_addr learned my addr 10.20.193.51:0/712329438 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): get_auth_request con 0x7f9d8c150220 auth_method 0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): get_auth_request con 0x7f9d8c15c710 auth_method 0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d917e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d83fff640 10 monclient(hunting): get_auth_request con 0x7f9d8c150600 auth_method 0 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d83fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d83fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.629+0000 7f9d90fe4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.630+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.630+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.630+0000 7f9d83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 10 monclient(hunting): handle_auth_done global_id 4987 payload 824 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 -- 10.20.193.51:0/712329438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d8c150600 msgr2=0x7f9d8c0c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d8c150600 0x7f9d8c0c1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 -- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 msgr2=0x7f9d8c1f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 10 monclient: _finish_hunting 0 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 monclient: found mon.noname-b 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d90fe4640 1 -- 10.20.193.51:0/712329438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c0c77e0 con 0x7f9d8c150220 2026-02-22T21:41:11.636 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.631+0000 7f9d917e5640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.632+0000 7f9d90fe4640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c500 tx=0x7f9d7c00c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.632+0000 7f9d817fa640 10 client.?.objecter ms_handle_connect 0x7f9d8c150220 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.632+0000 7f9d817fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c017070 con 0x7f9d8c150220 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: dump: 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:11.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 5 monclient: authenticate success, global_id 4987 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 10 monclient: _renew_subs 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8c0c4280 con 0x7f9d8c150220 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 10 client.4987.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 10 monclient: _renew_subs 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c0c7d30 con 0x7f9d8c150220 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d9326f640 1 librados: init done 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7c005de0 con 0x7f9d8c150220 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d82ffd640 4 set_mon_vals no callback set 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c021060 con 0x7f9d8c150220 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.633+0000 7f9d817fa640 10 monclient: dump: 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.634+0000 7f9d9326f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.634+0000 7f9d9326f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.634+0000 7f9d9326f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.634+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8c0c73a0 con 0x7f9d8c150220 2026-02-22T21:41:11.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.645+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9d7c024080 con 0x7f9d8c150220 2026-02-22T21:41:11.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.645+0000 7f9d817fa640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 0x7f9d5c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:11.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.645+0000 7f9d83fff640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 0x7f9d5c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:11.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.645+0000 7f9d83fff640 10 monclient: get_auth_request con 0x7f9d5c078e00 auth_method 0 2026-02-22T21:41:11.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.646+0000 7f9d817fa640 10 client.4987.objecter ms_handle_connect 0x7f9d5c078e00 2026-02-22T21:41:11.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.646+0000 7f9d83fff640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 0x7f9d5c07b2c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9d74039000 tx=0x7f9d74033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:11.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.647+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9d7c096e20 con 0x7f9d8c150220 2026-02-22T21:41:11.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.647+0000 7f9d817fa640 10 client.4987.objecter ms_dispatch2 0x7f9d8c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:11.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.647+0000 7f9d817fa640 3 client.4987.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:11.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.647+0000 7f9d817fa640 3 client.4987.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:11.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.647+0000 7f9d817fa640 20 client.4987.objecter dump_active .. 0 homeless 2026-02-22T21:41:11.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.750+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9d7c061450 con 0x7f9d8c150220 2026-02-22T21:41:11.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.750+0000 7f9d817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:11.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.750+0000 7f9d817fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:11.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.851+0000 7f9d9326f640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:11.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.851+0000 7f9d9326f640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:11.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.851+0000 7f9d9326f640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:11.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.851+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9d8c074da0 con 0x7f9d8c150220 2026-02-22T21:41:11.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.855+0000 7f9d817fa640 1 -- 10.20.193.51:0/712329438 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f9d7c065750 con 0x7f9d8c150220 2026-02-22T21:41:11.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.855+0000 7f9d817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:11.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.855+0000 7f9d817fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:11.856 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:11.856 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 10 librados: watch_flush enter 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 10 librados: watch_flush exit 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 20 client.4987.objecter shutdown clearing up homeless session... 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 10 client.4987.objecter successfully canceled tick 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 msgr2=0x7f9d5c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 0x7f9d5c07b2c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9d74039000 tx=0x7f9d74033000 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 10 monclient: shutdown 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 msgr2=0x7f9d8c0c3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c500 tx=0x7f9d7c00c9d0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 shutdown_connections 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9d5c078e00 0x7f9d5c07b2c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9d8c15c710 0x7f9d8c1f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9d8c150600 0x7f9d8c0c1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 --2- 10.20.193.51:0/712329438 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9d8c150220 0x7f9d8c0c3d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 >> 10.20.193.51:0/712329438 conn(0x7f9d8c0cd7b0 msgr2=0x7f9d8c0c0790 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 shutdown_connections 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 -- 10.20.193.51:0/712329438 wait complete. 2026-02-22T21:41:11.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:11.856+0000 7f9d9326f640 1 librados: shutdown 2026-02-22T21:41:14.868 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:14.938 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.936+0000 7f4d5680b640 10 monclient: get_monmap_and_config 2026-02-22T21:41:14.938 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.936+0000 7f4d5680b640 10 monclient: build_initial_monmap 2026-02-22T21:41:14.938 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.936+0000 7f4d5680b640 10 monclient: monmap: 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:14.937271+0000 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:14.937271+0000 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:14.939 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:14.940 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:14.940 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.936+0000 7f4d5680b640 1 Processor -- start 2026-02-22T21:41:14.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 1 -- start start 2026-02-22T21:41:14.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient: init 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient: _add_conns ranks=[0,5,1,2,4,3] 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d50150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4d5015c710 0x7f4d500c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d5015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f4d50150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d5015a920 con 0x7f4d50150220 2026-02-22T21:41:14.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d5015af60 con 0x7f4d50150600 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f4d5015b240 con 0x7f4d5015c710 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d5680b640 10 monclient(hunting): authenticate will time out at 1350.517672s 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d4ffff640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f4d50150220 0x7f4d5015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:36186/0 (socket says 10.20.193.51:36186) 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.937+0000 7f4d4ffff640 1 -- 10.20.193.51:0/3050134313 learned_addr learned my addr 10.20.193.51:0/3050134313 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 1 --2- 10.20.193.51:0/3050134313 >> v2:10.20.193.125:3300/0 conn(0x7f4d5015c710 0x7f4d500c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d5015c710 auth_method 0 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.938+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650361280 0 0) 0x7f4d5015a920 con 0x7f4d50150220 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _negotiate 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): my global_id is 4995 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d34003620 con 0x7f4d50150220 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028724717 0 0) 0x7f4d5015af60 con 0x7f4d50150600 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _negotiate 2026-02-22T21:41:14.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 10 monclient(hunting): my global_id is 4990 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d5015a920 con 0x7f4d50150600 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_done global_id 4955 payload 274 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/3050134313 >> v1:10.20.193.68:6789/0 conn(0x7f4d50150600 legacy=0x7f4d5006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/3050134313 >> v1:10.20.193.51:6789/0 conn(0x7f4d50150220 legacy=0x7f4d5015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 1 monclient: found mon.noname-c 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.939+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/3050134313 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5015b520 con 0x7f4d5015c710 2026-02-22T21:41:14.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.940+0000 7f4d4f7fe640 1 --2- 10.20.193.51:0/3050134313 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d5015c710 0x7f4d500c73a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098d0 tx=0x7f4d4002ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2820d222653fcb7 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4003c070 con 0x7f4d5015c710 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: dump: 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:14.956 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: _finish_auth 0 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: _check_auth_tickets 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4002fbf0 con 0x7f4d5015c710 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 5 monclient: authenticate success, global_id 4955 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 1 -- 10.20.193.51:0/3050134313 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40037440 con 0x7f4d5015c710 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:14.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 4 set_mon_vals no callback set 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d4effd640 10 monclient: dump: 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:14.958 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 10 monclient: shutdown 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 -- 10.20.193.51:0/3050134313 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d5015c710 msgr2=0x7f4d500c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 --2- 10.20.193.51:0/3050134313 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d5015c710 0x7f4d500c73a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098d0 tx=0x7f4d4002ef20 comp rx=0 tx=0).stop 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 -- 10.20.193.51:0/3050134313 shutdown_connections 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 --2- 10.20.193.51:0/3050134313 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d5015c710 0x7f4d500c73a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 -- 10.20.193.51:0/3050134313 >> 10.20.193.51:0/3050134313 conn(0x7f4d500cd7b0 msgr2=0x7f4d500cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 -- 10.20.193.51:0/3050134313 shutdown_connections 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.942+0000 7f4d5680b640 1 -- 10.20.193.51:0/3050134313 wait complete. 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 10 monclient: build_initial_monmap 2026-02-22T21:41:14.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 10 monclient: monmap: 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:14.960 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 librados: starting msgr at 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 librados: starting objecter 2026-02-22T21:41:14.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 Processor -- start 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 -- start start 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 librados: setting wanted keys 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 1 librados: calling monclient init 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 10 monclient: init 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.943+0000 7f4d5680b640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d50150220 0x7f4d5014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d50150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d50150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4d5015c710 0x7f4d501f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:14.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d5015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f4d500703b0 con 0x7f4d50150600 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5015bae0 con 0x7f4d5015c710 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50070530 con 0x7f4d50150220 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d5680b640 10 monclient(hunting): authenticate will time out at 1350.524038s 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:56868/0 (socket says 10.20.193.51:56868) 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/1720737163 learned_addr learned my addr 10.20.193.51:0/1720737163 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:14.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d50150600 auth_method 0 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d50150220 0x7f4d5014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4d5015c710 0x7f4d501f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): get_auth_request con 0x7f4d50150220 auth_method 0 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:14.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): get_auth_request con 0x7f4d5015c710 auth_method 0 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.944+0000 7f4d54d81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 10 monclient(hunting): handle_auth_done global_id 4998 payload 824 2026-02-22T21:41:14.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/1720737163 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d50150220 msgr2=0x7f4d5014a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d50150220 0x7f4d5014a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/1720737163 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4d5015c710 msgr2=0x7f4d501f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4d5015c710 0x7f4d501f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.946+0000 7f4d4f7fe640 1 -- 10.20.193.51:0/1720737163 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d500c8b10 con 0x7f4d50150600 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.947+0000 7f4d4f7fe640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098d0 tx=0x7f4d400047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.947+0000 7f4d4cff9640 10 client.?.objecter ms_handle_connect 0x7f4d50150600 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.947+0000 7f4d4cff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:14.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4003c040 con 0x7f4d50150600 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: dump: 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:14.967 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: _finish_auth 0 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: _check_auth_tickets 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d40042450 con 0x7f4d50150600 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40042820 con 0x7f4d50150600 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4e7fc640 4 set_mon_vals no callback set 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d4cff9640 10 monclient: dump: 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:14.968 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 5 monclient: authenticate success, global_id 4998 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 10 monclient: _renew_subs 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d500c8df0 con 0x7f4d50150600 2026-02-22T21:41:14.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 10 client.4998.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 10 monclient: _renew_subs 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d500c93b0 con 0x7f4d50150600 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.948+0000 7f4d5680b640 1 librados: init done 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.949+0000 7f4d5680b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.949+0000 7f4d5680b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.949+0000 7f4d5680b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:14.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:14.949+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d500c73a0 con 0x7f4d50150600 2026-02-22T21:41:15.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.055+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4d400429c0 con 0x7f4d50150600 2026-02-22T21:41:15.057 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.055+0000 7f4d4cff9640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 0x7f4d2007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:15.057 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.055+0000 7f4d4ffff640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 0x7f4d2007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:15.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.055+0000 7f4d4ffff640 10 monclient: get_auth_request con 0x7f4d20078e50 auth_method 0 2026-02-22T21:41:15.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.056+0000 7f4d4cff9640 10 client.4998.objecter ms_handle_connect 0x7f4d20078e50 2026-02-22T21:41:15.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.056+0000 7f4d4ffff640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 0x7f4d2007b310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c005980 tx=0x7f4d3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:15.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f4d4002fab0 con 0x7f4d50150600 2026-02-22T21:41:15.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 10 client.4998.objecter ms_dispatch2 0x7f4d50000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:15.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 3 client.4998.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:15.065 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 3 client.4998.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:15.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 20 client.4998.objecter dump_active .. 0 homeless 2026-02-22T21:41:15.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f4d400873d0 con 0x7f4d50150600 2026-02-22T21:41:15.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:15.066 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.063+0000 7f4d4cff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:15.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.168+0000 7f4d5680b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:15.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.168+0000 7f4d5680b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:15.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.168+0000 7f4d5680b640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:15.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.168+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4d500c4270 con 0x7f4d50150600 2026-02-22T21:41:15.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.171+0000 7f4d4cff9640 1 -- 10.20.193.51:0/1720737163 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f4d4008b6d0 con 0x7f4d50150600 2026-02-22T21:41:15.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.171+0000 7f4d4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:15.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.171+0000 7f4d4cff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:15.174 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:15.174 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 10 librados: watch_flush enter 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 10 librados: watch_flush exit 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 20 client.4998.objecter shutdown clearing up homeless session... 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 10 client.4998.objecter successfully canceled tick 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 msgr2=0x7f4d2007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 0x7f4d2007b310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c005980 tx=0x7f4d3c009040 comp rx=0 tx=0).stop 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 10 monclient: shutdown 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 msgr2=0x7f4d501f3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4d400098d0 tx=0x7f4d400047c0 comp rx=0 tx=0).stop 2026-02-22T21:41:15.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 shutdown_connections 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4d20078e50 0x7f4d2007b310 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4d5015c710 0x7f4d501f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4d50150600 0x7f4d501f3360 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 --2- 10.20.193.51:0/1720737163 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4d50150220 0x7f4d5014a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 >> 10.20.193.51:0/1720737163 conn(0x7f4d500cd7b0 msgr2=0x7f4d500cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 shutdown_connections 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 -- 10.20.193.51:0/1720737163 wait complete. 2026-02-22T21:41:15.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:15.172+0000 7f4d5680b640 1 librados: shutdown 2026-02-22T21:41:18.183 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:18.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.248+0000 7f66d88cd640 10 monclient: get_monmap_and_config 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.248+0000 7f66d88cd640 10 monclient: build_initial_monmap 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.248+0000 7f66d88cd640 10 monclient: monmap: 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:18.249528+0000 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:18.249528+0000 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:18.251 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 Processor -- start 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 -- start start 2026-02-22T21:41:18.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient: init 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient: _add_conns ranks=[3,0,2,5,4,1] 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f66d0150220 0x7f66d00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-a con 0x7f66d0150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66d0150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66d015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:18.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d015a920 con 0x7f66d0150600 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66d015af60 con 0x7f66d015c710 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f66d015b240 con 0x7f66d0150220 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.249+0000 7f66d88cd640 10 monclient(hunting): authenticate will time out at 1353.829827s 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6e43640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f66d015c710 0x7f66d006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:50518/0 (socket says 10.20.193.51:50518) 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6e43640 1 -- 10.20.193.51:0/1120303505 learned_addr learned my addr 10.20.193.51:0/1120303505 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6642640 1 --2- 10.20.193.51:0/1120303505 >> v2:10.20.193.51:3300/0 conn(0x7f66d0150220 0x7f66d00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6642640 10 monclient(hunting): get_auth_request con 0x7f66d0150220 auth_method 0 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6642640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:18.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.250+0000 7f66d6642640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 1 -- 10.20.193.51:0/1120303505 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206145034 0 0) 0x7f66d015af60 con 0x7f66d015c710 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 10 monclient(hunting): _negotiate 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 10 monclient(hunting): my global_id is 4961 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.251+0000 7f66d5640640 1 -- 10.20.193.51:0/1120303505 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66c0003620 con 0x7f66d015c710 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 10 monclient(hunting): handle_auth_done global_id 5001 payload 274 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 1 -- 10.20.193.51:0/1120303505 >> v1:10.20.193.125:6789/0 conn(0x7f66d015c710 legacy=0x7f66d006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 1 -- 10.20.193.51:0/1120303505 >> v1:10.20.193.51:6789/0 conn(0x7f66d0150600 legacy=0x7f66d015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 10 monclient: _finish_hunting 0 2026-02-22T21:41:18.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 1 monclient: found mon.noname-a 2026-02-22T21:41:18.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:18.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.252+0000 7f66d6642640 1 -- 10.20.193.51:0/1120303505 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d015b520 con 0x7f66d0150220 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.254+0000 7f66d6642640 1 --2- 10.20.193.51:0/1120303505 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d0150220 0x7f66d00c73a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563bade7f73f5c1d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 1 -- 10.20.193.51:0/1120303505 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c803c070 con 0x7f66d0150220 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: dump: 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.265 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: _finish_auth 0 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: _check_auth_tickets 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 1 -- 10.20.193.51:0/1120303505 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c802fc10 con 0x7f66d0150220 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 5 monclient: authenticate success, global_id 5001 2026-02-22T21:41:18.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 1 -- 10.20.193.51:0/1120303505 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8037440 con 0x7f66d0150220 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 4 set_mon_vals no callback set 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d5640640 10 monclient: dump: 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.267 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 10 monclient: shutdown 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 -- 10.20.193.51:0/1120303505 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d0150220 msgr2=0x7f66d00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1120303505 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d0150220 0x7f66d00c73a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c802ef00 comp rx=0 tx=0).stop 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 -- 10.20.193.51:0/1120303505 shutdown_connections 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1120303505 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d0150220 0x7f66d00c73a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 -- 10.20.193.51:0/1120303505 >> 10.20.193.51:0/1120303505 conn(0x7f66d00cd7b0 msgr2=0x7f66d00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 -- 10.20.193.51:0/1120303505 shutdown_connections 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.256+0000 7f66d88cd640 1 -- 10.20.193.51:0/1120303505 wait complete. 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient: build_initial_monmap 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient: monmap: 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:18.269 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 librados: starting msgr at 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 librados: starting objecter 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 Processor -- start 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 -- start start 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 librados: setting wanted keys 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 librados: calling monclient init 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient: init 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:18.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66d0150220 0x7f66d014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-c con 0x7f66d0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66d0150600 0x7f66d006ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-b con 0x7f66d0150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): picked mon.noname-a con 0x7f66d015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f66d00703b0 con 0x7f66d015c710 2026-02-22T21:41:18.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f66d015bae0 con 0x7f66d0150600 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f66d0070530 con 0x7f66d0150220 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.257+0000 7f66d88cd640 10 monclient(hunting): authenticate will time out at 1353.837834s 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:46842/0 (socket says 10.20.193.51:46842) 2026-02-22T21:41:18.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 1 -- 10.20.193.51:0/1159837618 learned_addr learned my addr 10.20.193.51:0/1159837618 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): get_auth_request con 0x7f66d015c710 auth_method 0 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66d0150220 0x7f66d014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:18.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6e43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d5e41640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66d0150600 0x7f66d006ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): get_auth_request con 0x7f66d0150220 auth_method 0 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d5e41640 10 monclient(hunting): get_auth_request con 0x7f66d0150600 auth_method 0 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d5e41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:18.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d5e41640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d5e41640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.258+0000 7f66d6642640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d5e41640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d5e41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d5e41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:18.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 10 monclient(hunting): handle_auth_done global_id 5007 payload 824 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 -- 10.20.193.51:0/1159837618 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66d0150220 msgr2=0x7f66d014ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66d0150220 0x7f66d014ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 -- 10.20.193.51:0/1159837618 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66d0150600 msgr2=0x7f66d006ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66d0150600 0x7f66d006ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 10 monclient: _finish_hunting 0 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 monclient: found mon.noname-a 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:18.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.259+0000 7f66d6e43640 1 -- 10.20.193.51:0/1159837618 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d0072400 con 0x7f66d015c710 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.260+0000 7f66d6e43640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f66c400cb70 tx=0x7f66c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.260+0000 7f66baffd640 10 client.?.objecter ms_handle_connect 0x7f66d015c710 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.260+0000 7f66baffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c400ad50 con 0x7f66d015c710 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:18.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: dump: 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:18.278 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: _finish_auth 0 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: _check_auth_tickets 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c4012400 con 0x7f66d015c710 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 5 monclient: authenticate success, global_id 5007 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _renew_subs 2026-02-22T21:41:18.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d00726e0 con 0x7f66d015c710 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c40127b0 con 0x7f66d015c710 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d4e3f640 4 set_mon_vals no callback set 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66baffd640 10 monclient: dump: 2026-02-22T21:41:18.280 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:18.281 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 client.5007.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _renew_subs 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d0072ca0 con 0x7f66d015c710 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 1 librados: init done 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:18.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:18.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.262+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d00c73a0 con 0x7f66d015c710 2026-02-22T21:41:18.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.274+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f66c4012950 con 0x7f66d015c710 2026-02-22T21:41:18.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.274+0000 7f66baffd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 0x7f66a007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:18.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.274+0000 7f66d6642640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 0x7f66a007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:18.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.274+0000 7f66d6642640 10 monclient: get_auth_request con 0x7f66a0078e50 auth_method 0 2026-02-22T21:41:18.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.275+0000 7f66d6642640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 0x7f66a007b310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66c8039000 tx=0x7f66c8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.275+0000 7f66baffd640 10 client.5007.objecter ms_handle_connect 0x7f66a0078e50 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.276+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f66c405e8d0 con 0x7f66d015c710 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.276+0000 7f66baffd640 10 client.5007.objecter ms_dispatch2 0x7f66d0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.276+0000 7f66baffd640 3 client.5007.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.276+0000 7f66baffd640 3 client.5007.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:18.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.276+0000 7f66baffd640 20 client.5007.objecter dump_active .. 0 homeless 2026-02-22T21:41:18.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.379+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f66c4062890 con 0x7f66d015c710 2026-02-22T21:41:18.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.380+0000 7f66baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:18.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.380+0000 7f66baffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:18.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.483+0000 7f66d88cd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:18.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.483+0000 7f66d88cd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:18.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.483+0000 7f66d88cd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:18.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.483+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f66d0074da0 con 0x7f66d015c710 2026-02-22T21:41:18.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.485+0000 7f66baffd640 1 -- 10.20.193.51:0/1159837618 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f66c4066b90 con 0x7f66d015c710 2026-02-22T21:41:18.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.485+0000 7f66baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:18.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.485+0000 7f66baffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:18.487 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:18.487 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:18.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 10 librados: watch_flush enter 2026-02-22T21:41:18.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 10 librados: watch_flush exit 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 20 client.5007.objecter shutdown clearing up homeless session... 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 10 client.5007.objecter successfully canceled tick 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 msgr2=0x7f66a007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 0x7f66a007b310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f66c8039000 tx=0x7f66c8033000 comp rx=0 tx=0).stop 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 10 monclient: shutdown 2026-02-22T21:41:18.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 msgr2=0x7f66d01f71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f66c400cb70 tx=0x7f66c400a430 comp rx=0 tx=0).stop 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 shutdown_connections 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f66a0078e50 0x7f66a007b310 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f66d015c710 0x7f66d01f71e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f66d0150600 0x7f66d006ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 --2- 10.20.193.51:0/1159837618 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f66d0150220 0x7f66d014ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:18.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 >> 10.20.193.51:0/1159837618 conn(0x7f66d00cd7b0 msgr2=0x7f66d00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:18.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 shutdown_connections 2026-02-22T21:41:18.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 -- 10.20.193.51:0/1159837618 wait complete. 2026-02-22T21:41:18.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:18.487+0000 7f66d88cd640 1 librados: shutdown 2026-02-22T21:41:21.498 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:21.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.567+0000 7f45331b3640 10 monclient: get_monmap_and_config 2026-02-22T21:41:21.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.567+0000 7f45331b3640 10 monclient: build_initial_monmap 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.567+0000 7f45331b3640 10 monclient: monmap: 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:21.569258+0000 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:21.569258+0000 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:21.571 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.567+0000 7f45331b3640 1 Processor -- start 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 -- start start 2026-02-22T21:41:21.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient: init 2026-02-22T21:41:21.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:21.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient: _add_conns ranks=[4,5,3,1,2,0] 2026-02-22T21:41:21.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f452c150220 0x7f452c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-b con 0x7f452c150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:21.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f452c150600 0x7f452c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-c con 0x7f452c150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f452c15c540 0x7f452c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-a con 0x7f452c15c540 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f452c15c9b0 con 0x7f452c15c540 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f452c15a920 con 0x7f452c150220 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f452c15af60 con 0x7f452c150600 2026-02-22T21:41:21.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f45331b3640 10 monclient(hunting): authenticate will time out at 1357.149550s 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.568+0000 7f4530f28640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f452c150220 0x7f452c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f452c150600 0x7f452c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f452c150220 0x7f452c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:39766/0 (socket says 10.20.193.51:39766) 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 1 -- 10.20.193.51:0/1378684531 learned_addr learned my addr 10.20.193.51:0/1378684531 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): get_auth_request con 0x7f452c150220 auth_method 0 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:21.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): get_auth_request con 0x7f452c150600 auth_method 0 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:21.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.569+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 10 monclient(hunting): handle_auth_done global_id 4999 payload 274 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 -- 10.20.193.51:0/1378684531 >> v2:10.20.193.125:3300/0 conn(0x7f452c150600 msgr2=0x7f452c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 --2- 10.20.193.51:0/1378684531 >> v2:10.20.193.125:3300/0 conn(0x7f452c150600 0x7f452c15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 -- 10.20.193.51:0/1378684531 >> v2:10.20.193.51:3300/0 conn(0x7f452c15c540 msgr2=0x7f452c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 --2- 10.20.193.51:0/1378684531 >> v2:10.20.193.51:3300/0 conn(0x7f452c15c540 0x7f452c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 10 monclient: _finish_hunting 0 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 monclient: found mon.noname-b 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:21.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.570+0000 7f4530f28640 1 -- 10.20.193.51:0/1378684531 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c15b240 con 0x7f452c150220 2026-02-22T21:41:21.580 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:52.727 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.580 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:52.729 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.578+0000 7f4530f28640 1 --2- 10.20.193.51:0/1378684531 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c0c73a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f452c070940 tx=0x7f451c02ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bec1c6ab4cb000ef server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 1 -- 10.20.193.51:0/1378684531 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c03c070 con 0x7f452c150220 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 10 monclient: dump: 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:21.582 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:21.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45331b3640 5 monclient: authenticate success, global_id 4999 2026-02-22T21:41:21.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.579+0000 7f45331b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:21.584 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:05:52.737 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.604 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:52.720 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.605 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:52.721 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.609 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:52.761 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 1 -- 10.20.193.51:0/1378684531 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f451c02fbd0 con 0x7f452c150220 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 1 -- 10.20.193.51:0/1378684531 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c037440 con 0x7f452c150220 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:21.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45237fe640 10 monclient: dump: 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45331b3640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.609+0000 7f45331b3640 4 set_mon_vals no callback set 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 10 monclient: shutdown 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 -- 10.20.193.51:0/1378684531 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 msgr2=0x7f452c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 --2- 10.20.193.51:0/1378684531 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c0c73a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f452c070940 tx=0x7f451c02ef50 comp rx=0 tx=0).stop 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 -- 10.20.193.51:0/1378684531 shutdown_connections 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 --2- 10.20.193.51:0/1378684531 >> v2:10.20.193.51:3300/0 conn(0x7f452c15c540 0x7f452c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 --2- 10.20.193.51:0/1378684531 >> v2:10.20.193.125:3300/0 conn(0x7f452c150600 0x7f452c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 --2- 10.20.193.51:0/1378684531 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c0c73a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 -- 10.20.193.51:0/1378684531 >> 10.20.193.51:0/1378684531 conn(0x7f452c0cd7b0 msgr2=0x7f452c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:21.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 -- 10.20.193.51:0/1378684531 shutdown_connections 2026-02-22T21:41:21.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.610+0000 7f45331b3640 1 -- 10.20.193.51:0/1378684531 wait complete. 2026-02-22T21:41:21.616 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:52.764 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.617 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:05:52.728 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.619 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 10 monclient: build_initial_monmap 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 10 monclient: monmap: 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:21.620 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 librados: starting msgr at 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 librados: starting objecter 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 Processor -- start 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 -- start start 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 librados: setting wanted keys 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 1 librados: calling monclient init 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 10 monclient: init 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.611+0000 7f45331b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:21.621 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c1f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-b con 0x7f452c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-c con 0x7f452c150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f452c15c540 0x7f452c1e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): picked mon.noname-a con 0x7f452c15c540 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f452c15bae0 con 0x7f452c15c540 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f452c15b800 con 0x7f452c150220 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0701e0 con 0x7f452c150600 2026-02-22T21:41:21.622 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.612+0000 7f45331b3640 10 monclient(hunting): authenticate will time out at 1357.193050s 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.613+0000 7f4523fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4530f28640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c1f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:43436/0 (socket says 10.20.193.51:43436) 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 1 -- 10.20.193.51:0/3689229472 learned_addr learned my addr 10.20.193.51:0/3689229472 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 10 monclient(hunting): get_auth_request con 0x7f452c150600 auth_method 0 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.614+0000 7f4523fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.615+0000 7f4530f28640 10 monclient(hunting): get_auth_request con 0x7f452c150220 auth_method 0 2026-02-22T21:41:21.623 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.615+0000 7f4530f28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:21.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.615+0000 7f4530f28640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:21.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.615+0000 7f4530f28640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:21.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.616+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:21.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.616+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:21.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.616+0000 7f4523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:21.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.618+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:21.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.618+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.618+0000 7f4530f28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 10 monclient(hunting): handle_auth_done global_id 4970 payload 824 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 -- 10.20.193.51:0/3689229472 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 msgr2=0x7f452c1f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c1f7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 -- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f452c15c540 msgr2=0x7f452c1e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f452c15c540 0x7f452c1e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 monclient: found mon.noname-c 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:41:21.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.619+0000 7f4523fff640 1 -- 10.20.193.51:0/3689229472 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c073b50 con 0x7f452c150600 2026-02-22T21:41:21.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.622+0000 7f4523fff640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f451400ec10 tx=0x7f451400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:21.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.622+0000 7f45217fa640 10 client.?.objecter ms_handle_connect 0x7f452c150600 2026-02-22T21:41:21.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.622+0000 7f45217fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:21.627 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:05:52.777 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514017070 con 0x7f452c150600 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: dump: 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.630 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f451400cc40 con 0x7f452c150600 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:21.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514013470 con 0x7f452c150600 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45217fa640 10 monclient: dump: 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:21.632 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f4522ffd640 4 set_mon_vals no callback set 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 5 monclient: authenticate success, global_id 4970 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 10 monclient: _renew_subs 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c073e30 con 0x7f452c150600 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 10 client.4970.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 10 monclient: _renew_subs 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c0743f0 con 0x7f452c150600 2026-02-22T21:41:21.633 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.626+0000 7f45331b3640 1 librados: init done 2026-02-22T21:41:21.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.627+0000 7f45331b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:21.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.627+0000 7f45331b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:21.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.627+0000 7f45331b3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:21.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.627+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c15e110 con 0x7f452c150600 2026-02-22T21:41:21.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.737+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4514013630 con 0x7f452c150600 2026-02-22T21:41:21.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.737+0000 7f45217fa640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 0x7f44fc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:21.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.737+0000 7f4530f28640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 0x7f44fc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:21.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.737+0000 7f4530f28640 10 monclient: get_auth_request con 0x7f44fc078e00 auth_method 0 2026-02-22T21:41:21.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.738+0000 7f4530f28640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 0x7f44fc07b2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f452c0c65c0 tx=0x7f451c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:21.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.738+0000 7f45217fa640 10 client.4970.objecter ms_handle_connect 0x7f44fc078e00 2026-02-22T21:41:21.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.741+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f451405e560 con 0x7f452c150600 2026-02-22T21:41:21.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.741+0000 7f45217fa640 10 client.4970.objecter ms_dispatch2 0x7f452c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:21.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.741+0000 7f45217fa640 3 client.4970.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:21.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.741+0000 7f45217fa640 3 client.4970.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:21.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.741+0000 7f45217fa640 20 client.4970.objecter dump_active .. 0 homeless 2026-02-22T21:41:21.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.748+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f452c15e110 con 0x7f452c150600 2026-02-22T21:41:21.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.748+0000 7f45217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:21.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.748+0000 7f45217fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:21.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.849+0000 7f45331b3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:21.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.849+0000 7f45331b3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:21.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.849+0000 7f45331b3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:21.852 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.849+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f452c074da0 con 0x7f452c150600 2026-02-22T21:41:21.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.852+0000 7f45217fa640 1 -- 10.20.193.51:0/3689229472 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f4514062520 con 0x7f452c150600 2026-02-22T21:41:21.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.852+0000 7f45217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:21.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.852+0000 7f45217fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:21.855 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:21.855 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 10 librados: watch_flush enter 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 10 librados: watch_flush exit 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 20 client.4970.objecter shutdown clearing up homeless session... 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 10 client.4970.objecter successfully canceled tick 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 msgr2=0x7f44fc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 0x7f44fc07b2c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f452c0c65c0 tx=0x7f451c033000 comp rx=0 tx=0).stop 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 10 monclient: shutdown 2026-02-22T21:41:21.857 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 msgr2=0x7f452c14dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f451400ec10 tx=0x7f451400c520 comp rx=0 tx=0).stop 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 shutdown_connections 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f44fc078e00 0x7f44fc07b2c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f452c15c540 0x7f452c1e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f452c150600 0x7f452c14dcb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 --2- 10.20.193.51:0/3689229472 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f452c150220 0x7f452c1f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 >> 10.20.193.51:0/3689229472 conn(0x7f452c0cd7b0 msgr2=0x7f452c0bfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 shutdown_connections 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 -- 10.20.193.51:0/3689229472 wait complete. 2026-02-22T21:41:21.858 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:21.853+0000 7f45331b3640 1 librados: shutdown 2026-02-22T21:41:24.865 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:24.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 10 monclient: get_monmap_and_config 2026-02-22T21:41:24.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 10 monclient: build_initial_monmap 2026-02-22T21:41:24.934 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 10 monclient: monmap: 2026-02-22T21:41:24.934 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:24.934 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:24.932953+0000 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:24.932953+0000 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.935 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 1 Processor -- start 2026-02-22T21:41:24.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 1 -- start start 2026-02-22T21:41:24.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.931+0000 7f1a42ef3640 10 monclient: init 2026-02-22T21:41:24.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient: _add_conns ranks=[3,4,5,1,0,2] 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1a3c150220 0x7f1a3c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a3c150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1a3c150600 0x7f1a3c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3c150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f1a3c15c540 0x7f1a3c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a3c15c540 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3c15c9b0 con 0x7f1a3c150220 2026-02-22T21:41:24.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3c15a920 con 0x7f1a3c150600 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3c15af60 con 0x7f1a3c15c540 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a42ef3640 10 monclient(hunting): authenticate will time out at 1360.513229s 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1a3c150220 0x7f1a3c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a33fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1a3c150600 0x7f1a3c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1a3c150220 0x7f1a3c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:46870/0 (socket says 10.20.193.51:46870) 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 1 -- 10.20.193.51:0/645971280 learned_addr learned my addr 10.20.193.51:0/645971280 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 10 monclient(hunting): get_auth_request con 0x7f1a3c150220 auth_method 0 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a40c68640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a33fff640 10 monclient(hunting): get_auth_request con 0x7f1a3c150600 auth_method 0 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a33fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.932+0000 7f1a33fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 1 --2- 10.20.193.51:0/645971280 >> v2:10.20.193.125:3300/0 conn(0x7f1a3c15c540 0x7f1a3c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): get_auth_request con 0x7f1a3c15c540 auth_method 0 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.933+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 10 monclient(hunting): handle_auth_done global_id 5010 payload 274 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 -- 10.20.193.51:0/645971280 >> v2:10.20.193.125:3300/0 conn(0x7f1a3c15c540 msgr2=0x7f1a3c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 --2- 10.20.193.51:0/645971280 >> v2:10.20.193.125:3300/0 conn(0x7f1a3c15c540 0x7f1a3c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 -- 10.20.193.51:0/645971280 >> v2:10.20.193.68:3300/0 conn(0x7f1a3c150600 msgr2=0x7f1a3c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 --2- 10.20.193.51:0/645971280 >> v2:10.20.193.68:3300/0 conn(0x7f1a3c150600 0x7f1a3c15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 10 monclient: _finish_hunting 0 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 monclient: found mon.noname-a 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.934+0000 7f1a40c68640 1 -- 10.20.193.51:0/645971280 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c15b240 con 0x7f1a3c150220 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.935+0000 7f1a40c68640 1 --2- 10.20.193.51:0/645971280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150220 0x7f1a3c0c73a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c074940 tx=0x7f1a2c02ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fe2e7b2d456a2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 1 -- 10.20.193.51:0/645971280 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c03c070 con 0x7f1a3c150220 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: dump: 2026-02-22T21:41:24.949 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:24.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a42ef3640 5 monclient: authenticate success, global_id 5010 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 1 -- 10.20.193.51:0/645971280 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a2c02fbd0 con 0x7f1a3c150220 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 1 -- 10.20.193.51:0/645971280 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c037440 con 0x7f1a3c150220 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a337fe640 10 monclient: dump: 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.951 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a42ef3640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.936+0000 7f1a42ef3640 4 set_mon_vals no callback set 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 10 monclient: shutdown 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- 10.20.193.51:0/645971280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150220 msgr2=0x7f1a3c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/645971280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150220 0x7f1a3c0c73a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c074940 tx=0x7f1a2c02ef50 comp rx=0 tx=0).stop 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- 10.20.193.51:0/645971280 shutdown_connections 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/645971280 >> v2:10.20.193.125:3300/0 conn(0x7f1a3c15c540 0x7f1a3c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/645971280 >> v2:10.20.193.68:3300/0 conn(0x7f1a3c150600 0x7f1a3c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/645971280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150220 0x7f1a3c0c73a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- 10.20.193.51:0/645971280 >> 10.20.193.51:0/645971280 conn(0x7f1a3c0cd7b0 msgr2=0x7f1a3c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- 10.20.193.51:0/645971280 shutdown_connections 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- 10.20.193.51:0/645971280 wait complete. 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 10 monclient: build_initial_monmap 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 10 monclient: monmap: 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:24.953 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 librados: starting msgr at 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 librados: starting objecter 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 Processor -- start 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 -- start start 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 librados: setting wanted keys 2026-02-22T21:41:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 1 librados: calling monclient init 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.937+0000 7f1a42ef3640 10 monclient: init 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1a3c150220 0x7f1a3c1f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-c con 0x7f1a3c150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-a con 0x7f1a3c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1a3c15c540 0x7f1a3c14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3c15c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c15bae0 con 0x7f1a3c150600 2026-02-22T21:41:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c15b800 con 0x7f1a3c15c540 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c0701e0 con 0x7f1a3c150220 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a42ef3640 10 monclient(hunting): authenticate will time out at 1360.519546s 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:46882/0 (socket says 10.20.193.51:46882) 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 1 -- 10.20.193.51:0/757396280 learned_addr learned my addr 10.20.193.51:0/757396280 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 10 monclient(hunting): get_auth_request con 0x7f1a3c150600 auth_method 0 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a33fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a40c68640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1a3c150220 0x7f1a3c1f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.938+0000 7f1a41469640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1a3c15c540 0x7f1a3c14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): get_auth_request con 0x7f1a3c150220 auth_method 0 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): get_auth_request con 0x7f1a3c15c540 auth_method 0 2026-02-22T21:41:24.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a40c68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.939+0000 7f1a41469640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 10 monclient(hunting): handle_auth_done global_id 5013 payload 824 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 -- 10.20.193.51:0/757396280 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1a3c150220 msgr2=0x7f1a3c1f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1a3c150220 0x7f1a3c1f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 -- 10.20.193.51:0/757396280 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1a3c15c540 msgr2=0x7f1a3c14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1a3c15c540 0x7f1a3c14c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 monclient: found mon.noname-a 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.940+0000 7f1a33fff640 1 -- 10.20.193.51:0/757396280 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c0c1e40 con 0x7f1a3c150600 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.941+0000 7f1a33fff640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400ebe0 tx=0x7f1a2400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.941+0000 7f1a317fa640 10 client.?.objecter ms_handle_connect 0x7f1a3c150600 2026-02-22T21:41:24.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.941+0000 7f1a317fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24017070 con 0x7f1a3c150600 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: dump: 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.962 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 5 monclient: authenticate success, global_id 5013 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a2400cc40 con 0x7f1a3c150600 2026-02-22T21:41:24.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24013470 con 0x7f1a3c150600 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a32ffd640 4 set_mon_vals no callback set 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a317fa640 10 monclient: dump: 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:24.964 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _renew_subs 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c157000 con 0x7f1a3c150600 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 client.5013.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _renew_subs 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c1f18a0 con 0x7f1a3c150600 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 1 librados: init done 2026-02-22T21:41:24.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.943+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3c070570 con 0x7f1a3c150600 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.955+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1a24024060 con 0x7f1a3c150600 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.955+0000 7f1a317fa640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 0x7f1a0c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.955+0000 7f1a40c68640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 0x7f1a0c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.955+0000 7f1a40c68640 10 monclient: get_auth_request con 0x7f1a0c078e80 auth_method 0 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.956+0000 7f1a317fa640 10 client.5013.objecter ms_handle_connect 0x7f1a0c078e80 2026-02-22T21:41:24.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:24.956+0000 7f1a40c68640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 0x7f1a0c07b340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c070940 tx=0x7f1a2c039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:25.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f1a2405e1e0 con 0x7f1a3c150600 2026-02-22T21:41:25.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 10 client.5013.objecter ms_dispatch2 0x7f1a3c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 3 client.5013.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 3 client.5013.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 20 client.5013.objecter dump_active .. 0 homeless 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.056+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f1a240621a0 con 0x7f1a3c150600 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.057+0000 7f1a317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:25.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.057+0000 7f1a317fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:25.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.157+0000 7f1a42ef3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:25.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.157+0000 7f1a42ef3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:25.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.157+0000 7f1a42ef3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:25.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.157+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1a3c1568a0 con 0x7f1a3c150600 2026-02-22T21:41:25.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.160+0000 7f1a317fa640 1 -- 10.20.193.51:0/757396280 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f1a240664a0 con 0x7f1a3c150600 2026-02-22T21:41:25.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.160+0000 7f1a317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:25.163 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.160+0000 7f1a317fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:25.164 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:25.164 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:25.165 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 10 librados: watch_flush enter 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 10 librados: watch_flush exit 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 20 client.5013.objecter shutdown clearing up homeless session... 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 10 client.5013.objecter successfully canceled tick 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 msgr2=0x7f1a0c07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 0x7f1a0c07b340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c070940 tx=0x7f1a2c039040 comp rx=0 tx=0).stop 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 10 monclient: shutdown 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 msgr2=0x7f1a3c156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400ebe0 tx=0x7f1a2400c520 comp rx=0 tx=0).stop 2026-02-22T21:41:25.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 shutdown_connections 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1a0c078e80 0x7f1a0c07b340 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1a3c15c540 0x7f1a3c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1a3c150600 0x7f1a3c156ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 --2- 10.20.193.51:0/757396280 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1a3c150220 0x7f1a3c1f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 >> 10.20.193.51:0/757396280 conn(0x7f1a3c0cd7b0 msgr2=0x7f1a3c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 shutdown_connections 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 -- 10.20.193.51:0/757396280 wait complete. 2026-02-22T21:41:25.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:25.161+0000 7f1a42ef3640 1 librados: shutdown 2026-02-22T21:41:28.174 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:28.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 10 monclient: get_monmap_and_config 2026-02-22T21:41:28.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 10 monclient: build_initial_monmap 2026-02-22T21:41:28.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 10 monclient: monmap: 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:28.243039+0000 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:28.243039+0000 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:28.245 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:28.246 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 1 Processor -- start 2026-02-22T21:41:28.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 1 -- start start 2026-02-22T21:41:28.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.241+0000 7fd27b14d640 10 monclient: init 2026-02-22T21:41:28.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:28.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient: _add_conns ranks=[5,2,4,0,1,3] 2026-02-22T21:41:28.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd274152ab0 0x7fd2740c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-c con 0x7fd274152ab0 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd27406d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd2740c95a0 0x7fd274158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2740c95a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27406f1d0 con 0x7fd27406d9d0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd27406ed40 con 0x7fd2740c95a0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd27406de50 con 0x7fd274152ab0 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:28.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd27b14d640 10 monclient(hunting): authenticate will time out at 1363.823335s 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd2796c3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd2740c95a0 0x7fd274158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.242+0000 7fd278ec2640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd274152ab0 0x7fd2740c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd2740c95a0 0x7fd274158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47430/0 (socket says 10.20.193.51:47430) 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 1 -- 10.20.193.51:0/764934542 learned_addr learned my addr 10.20.193.51:0/764934542 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): get_auth_request con 0x7fd2740c95a0 auth_method 0 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): get_auth_request con 0x7fd274152ab0 auth_method 0 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:28.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.243+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 10 monclient(hunting): handle_auth_done global_id 5011 payload 274 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 1 -- 10.20.193.51:0/764934542 >> v2:10.20.193.125:3300/0 conn(0x7fd274152ab0 msgr2=0x7fd2740c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 1 --2- 10.20.193.51:0/764934542 >> v2:10.20.193.125:3300/0 conn(0x7fd274152ab0 0x7fd2740c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 1 -- 10.20.193.51:0/764934542 >> v1:10.20.193.125:6789/0 conn(0x7fd27406d9d0 legacy=0x7fd2740c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 10 monclient: _finish_hunting 0 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 1 monclient: found mon.noname-b 2026-02-22T21:41:28.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:28.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:28.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.244+0000 7fd2796c3640 1 -- 10.20.193.51:0/764934542 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27406e130 con 0x7fd2740c95a0 2026-02-22T21:41:28.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.245+0000 7fd2796c3640 1 --2- 10.20.193.51:0/764934542 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2740c95a0 0x7fd274158a40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd270009e50 tx=0x7fd27002f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fc885f68cc1a9d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 1 -- 10.20.193.51:0/764934542 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27002fdf0 con 0x7fd2740c95a0 2026-02-22T21:41:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: dump: 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 1 -- 10.20.193.51:0/764934542 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2700373f0 con 0x7fd2740c95a0 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 1 -- 10.20.193.51:0/764934542 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700377c0 con 0x7fd2740c95a0 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd26b7fe640 10 monclient: dump: 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:28.260 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:28.261 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 5 monclient: authenticate success, global_id 5011 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 4 set_mon_vals no callback set 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 10 monclient: shutdown 2026-02-22T21:41:28.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 -- 10.20.193.51:0/764934542 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2740c95a0 msgr2=0x7fd274158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 --2- 10.20.193.51:0/764934542 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2740c95a0 0x7fd274158a40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd270009e50 tx=0x7fd27002f330 comp rx=0 tx=0).stop 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 -- 10.20.193.51:0/764934542 shutdown_connections 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 --2- 10.20.193.51:0/764934542 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd2740c95a0 0x7fd274158a40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 --2- 10.20.193.51:0/764934542 >> v2:10.20.193.125:3300/0 conn(0x7fd274152ab0 0x7fd2740c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 -- 10.20.193.51:0/764934542 >> 10.20.193.51:0/764934542 conn(0x7fd2740cd7b0 msgr2=0x7fd2740cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 -- 10.20.193.51:0/764934542 shutdown_connections 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.247+0000 7fd27b14d640 1 -- 10.20.193.51:0/764934542 wait complete. 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 10 monclient: build_initial_monmap 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 10 monclient: monmap: 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:28.264 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:28.265 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:28.266 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:28.266 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 librados: starting msgr at 2026-02-22T21:41:28.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 librados: starting objecter 2026-02-22T21:41:28.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 Processor -- start 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 -- start start 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 librados: setting wanted keys 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 1 librados: calling monclient init 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.248+0000 7fd27b14d640 10 monclient: init 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:28.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-a con 0x7fd27406d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2740c95a0 0x7fd27415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2740c95a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd274152ab0 0x7fd2740736c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): picked mon.noname-b con 0x7fd274152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd27406e9d0 con 0x7fd27406d9d0 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd27406e6f0 con 0x7fd274152ab0 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:28.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd274152e90 con 0x7fd2740c95a0 2026-02-22T21:41:28.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:28.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd27b14d640 10 monclient(hunting): authenticate will time out at 1363.830278s 2026-02-22T21:41:28.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd26bfff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2740c95a0 0x7fd27415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48136/0 (socket says 10.20.193.51:48136) 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 1 -- 10.20.193.51:0/1453735736 learned_addr learned my addr 10.20.193.51:0/1453735736 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 10 monclient(hunting): get_auth_request con 0x7fd27406d9d0 auth_method 0 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd278ec2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:28.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd2796c3640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd274152ab0 0x7fd2740736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd26bfff640 10 monclient(hunting): get_auth_request con 0x7fd2740c95a0 auth_method 0 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd26bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd26bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd2796c3640 10 monclient(hunting): get_auth_request con 0x7fd274152ab0 auth_method 0 2026-02-22T21:41:28.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd2796c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:28.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd2796c3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:28.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.249+0000 7fd2796c3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:28.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:28.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:28.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd278ec2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:28.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.250+0000 7fd2796c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 10 monclient(hunting): handle_auth_done global_id 5016 payload 824 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 -- 10.20.193.51:0/1453735736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2740c95a0 msgr2=0x7fd27415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2740c95a0 0x7fd27415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 -- 10.20.193.51:0/1453735736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd274152ab0 msgr2=0x7fd2740736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd274152ab0 0x7fd2740736c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 10 monclient: _finish_hunting 0 2026-02-22T21:41:28.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 monclient: found mon.noname-a 2026-02-22T21:41:28.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:28.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:28.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.251+0000 7fd278ec2640 1 -- 10.20.193.51:0/1453735736 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27415ace0 con 0x7fd27406d9d0 2026-02-22T21:41:28.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.252+0000 7fd278ec2640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd26400e890 tx=0x7fd26400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.252+0000 7fd2697fa640 10 client.?.objecter ms_handle_connect 0x7fd27406d9d0 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.252+0000 7fd2697fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264017070 con 0x7fd27406d9d0 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:28.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: dump: 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26400c940 con 0x7fd27406d9d0 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26400cd30 con 0x7fd27406d9d0 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 5 monclient: authenticate success, global_id 5016 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd26affd640 4 set_mon_vals no callback set 2026-02-22T21:41:28.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _renew_subs 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd274159e80 con 0x7fd27406d9d0 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd2697fa640 10 monclient: dump: 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:28.280 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 client.5016.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _renew_subs 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27415a440 con 0x7fd27406d9d0 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 1 librados: init done 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:28.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.254+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2740c8e90 con 0x7fd27406d9d0 2026-02-22T21:41:28.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd264024060 con 0x7fd27406d9d0 2026-02-22T21:41:28.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd2697fa640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 0x7fd24407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:28.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd26bfff640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 0x7fd24407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:28.364 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd26bfff640 10 monclient: get_auth_request con 0x7fd244078e00 auth_method 0 2026-02-22T21:41:28.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd2697fa640 10 client.5016.objecter ms_handle_connect 0x7fd244078e00 2026-02-22T21:41:28.365 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.362+0000 7fd26bfff640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 0x7fd24407b2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd2740c6ec0 tx=0x7fd25c007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd26405e0d0 con 0x7fd27406d9d0 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 10 client.5016.objecter ms_dispatch2 0x7fd274000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 3 client.5016.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 3 client.5016.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 20 client.5016.objecter dump_active .. 0 homeless 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd264062090 con 0x7fd27406d9d0 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:28.373 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.370+0000 7fd2697fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:28.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.469+0000 7fd27b14d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:28.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.469+0000 7fd27b14d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:28.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.469+0000 7fd27b14d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:28.473 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.469+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd27415c1a0 con 0x7fd27406d9d0 2026-02-22T21:41:28.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd2697fa640 1 -- 10.20.193.51:0/1453735736 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fd264066390 con 0x7fd27406d9d0 2026-02-22T21:41:28.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd2697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:28.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd2697fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:28.475 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:28.475 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 10 librados: watch_flush enter 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 10 librados: watch_flush exit 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 20 client.5016.objecter shutdown clearing up homeless session... 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 10 client.5016.objecter successfully canceled tick 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 msgr2=0x7fd24407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 0x7fd24407b2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd2740c6ec0 tx=0x7fd25c007f30 comp rx=0 tx=0).stop 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 10 monclient: shutdown 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 msgr2=0x7fd2741f43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:28.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd26400e890 tx=0x7fd26400ed60 comp rx=0 tx=0).stop 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 shutdown_connections 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd244078e00 0x7fd24407b2c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd274152ab0 0x7fd2740736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd2740c95a0 0x7fd27415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 --2- 10.20.193.51:0/1453735736 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd27406d9d0 0x7fd2741f43d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 >> 10.20.193.51:0/1453735736 conn(0x7fd2740cd7b0 msgr2=0x7fd2740c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 shutdown_connections 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.472+0000 7fd27b14d640 1 -- 10.20.193.51:0/1453735736 wait complete. 2026-02-22T21:41:28.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:28.473+0000 7fd27b14d640 1 librados: shutdown 2026-02-22T21:41:31.484 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.551+0000 7f5ee9c1a640 10 monclient: get_monmap_and_config 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.551+0000 7f5ee9c1a640 10 monclient: build_initial_monmap 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.551+0000 7f5ee9c1a640 10 monclient: monmap: 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:31.553569+0000 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:31.553569+0000 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:31.554 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 Processor -- start 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 -- start start 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient: init 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient: _add_conns ranks=[1,3,4,5,0,2] 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f5ee4152ab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5ee406d9d0 0x7f5ee40c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee406d9d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5ee40c93d0 0x7f5ee4158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee40c93d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ee406f1d0 con 0x7f5ee4152ab0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee406ed40 con 0x7f5ee406d9d0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee40c9880 con 0x7f5ee40c93d0 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:31.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.552+0000 7f5ee9c1a640 10 monclient(hunting): authenticate will time out at 1367.133962s 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5ee406d9d0 0x7f5ee40c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f5ee406d9d0 0x7f5ee40c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48144/0 (socket says 10.20.193.51:48144) 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/2837477947 learned_addr learned my addr 10.20.193.51:0/2837477947 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): get_auth_request con 0x7f5ee406d9d0 auth_method 0 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.553+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee3fff640 1 --2- 10.20.193.51:0/2837477947 >> v2:10.20.193.68:3300/0 conn(0x7f5ee40c93d0 0x7f5ee4158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee3fff640 10 monclient(hunting): get_auth_request con 0x7f5ee40c93d0 auth_method 0 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:31.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_done global_id 5019 payload 274 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/2837477947 >> v2:10.20.193.68:3300/0 conn(0x7f5ee40c93d0 msgr2=0x7f5ee4158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 1 --2- 10.20.193.51:0/2837477947 >> v2:10.20.193.68:3300/0 conn(0x7f5ee40c93d0 0x7f5ee4158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/2837477947 >> v1:10.20.193.68:6789/0 conn(0x7f5ee4152ab0 legacy=0x7f5ee40c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 1 monclient: found mon.noname-a 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.554+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/2837477947 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee406de50 con 0x7f5ee406d9d0 2026-02-22T21:41:31.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.555+0000 7f5ee2ffd640 1 --2- 10.20.193.51:0/2837477947 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee406d9d0 0x7f5ee40c8e90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009970 tx=0x7f5ed402efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647aff567c38c1d1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 1 -- 10.20.193.51:0/2837477947 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed403c070 con 0x7f5ee406d9d0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: dump: 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: _finish_auth 0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: _check_auth_tickets 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 1 -- 10.20.193.51:0/2837477947 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed402fc80 con 0x7f5ee406d9d0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 5 monclient: authenticate success, global_id 5019 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 1 -- 10.20.193.51:0/2837477947 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4037660 con 0x7f5ee406d9d0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee27fc640 10 monclient: dump: 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:31.560 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 4 set_mon_vals no callback set 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 10 monclient: shutdown 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/2837477947 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee406d9d0 msgr2=0x7f5ee40c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/2837477947 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee406d9d0 0x7f5ee40c8e90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009970 tx=0x7f5ed402efb0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/2837477947 shutdown_connections 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/2837477947 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee406d9d0 0x7f5ee40c8e90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/2837477947 >> v2:10.20.193.68:3300/0 conn(0x7f5ee40c93d0 0x7f5ee4158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/2837477947 >> 10.20.193.51:0/2837477947 conn(0x7f5ee40cd7b0 msgr2=0x7f5ee40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/2837477947 shutdown_connections 2026-02-22T21:41:31.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.557+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/2837477947 wait complete. 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 10 monclient: build_initial_monmap 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 10 monclient: monmap: 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 librados: starting msgr at 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 librados: starting objecter 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 Processor -- start 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 -- start start 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 librados: setting wanted keys 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 1 librados: calling monclient init 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.558+0000 7f5ee9c1a640 10 monclient: init 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5ee406d9d0 0x7f5ee4073de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee406d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee40c93d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5ee4152ab0 0x7f5ee415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee4152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee406e6f0 con 0x7f5ee40c93d0 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee406e410 con 0x7f5ee4152ab0 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee406e9d0 con 0x7f5ee406d9d0 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:31.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee9c1a640 10 monclient(hunting): authenticate will time out at 1367.140277s 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee3fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5ee4152ab0 0x7f5ee415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:48156/0 (socket says 10.20.193.51:48156) 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/3883043842 learned_addr learned my addr 10.20.193.51:0/3883043842 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): get_auth_request con 0x7f5ee40c93d0 auth_method 0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee3fff640 10 monclient(hunting): get_auth_request con 0x7f5ee4152ab0 auth_method 0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.559+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.560+0000 7f5ee3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.560+0000 7f5ee3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.560+0000 7f5ee3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 10 monclient(hunting): handle_auth_done global_id 5022 payload 824 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/3883043842 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5ee406d9d0 msgr2=0x7f5ee4073de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5ee406d9d0 0x7f5ee4073de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/3883043842 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5ee4152ab0 msgr2=0x7f5ee415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5ee4152ab0 0x7f5ee415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 monclient: found mon.noname-a 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.561+0000 7f5ee2ffd640 1 -- 10.20.193.51:0/3883043842 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee415a140 con 0x7f5ee40c93d0 2026-02-22T21:41:31.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.562+0000 7f5ee2ffd640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009aa0 tx=0x7f5ed40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:31.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.562+0000 7f5ebffff640 10 client.?.objecter ms_handle_connect 0x7f5ee40c93d0 2026-02-22T21:41:31.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.562+0000 7f5ebffff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed403c040 con 0x7f5ee40c93d0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: dump: 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: _finish_auth 0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed4004230 con 0x7f5ee40c93d0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4004620 con 0x7f5ee40c93d0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ee1ffb640 4 set_mon_vals no callback set 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ebffff640 10 monclient: dump: 2026-02-22T21:41:31.566 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ee9c1a640 5 monclient: authenticate success, global_id 5022 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ee9c1a640 10 monclient: _renew_subs 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ee9c1a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.563+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee4148a80 con 0x7f5ee40c93d0 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 client.5022.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 monclient: _renew_subs 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee4149040 con 0x7f5ee40c93d0 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 1 librados: init done 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.564+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee4152e90 con 0x7f5ee40c93d0 2026-02-22T21:41:31.671 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f5ed402fb00 con 0x7f5ee40c93d0 2026-02-22T21:41:31.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ebffff640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 0x7f5eb407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:31.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ee37fe640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 0x7f5eb407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:31.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ee37fe640 10 monclient: get_auth_request con 0x7f5eb4078e00 auth_method 0 2026-02-22T21:41:31.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ee37fe640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 0x7f5eb407b2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0002770 tx=0x7f5ed0007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:31.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.669+0000 7f5ebffff640 10 client.5022.objecter ms_handle_connect 0x7f5eb4078e00 2026-02-22T21:41:31.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f5ed4082a10 con 0x7f5ee40c93d0 2026-02-22T21:41:31.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 10 client.5022.objecter ms_dispatch2 0x7f5ee4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:31.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 3 client.5022.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:31.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 3 client.5022.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:31.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 20 client.5022.objecter dump_active .. 0 homeless 2026-02-22T21:41:31.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f5ed40870d0 con 0x7f5ee40c93d0 2026-02-22T21:41:31.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:31.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.683+0000 7f5ebffff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:31.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.784+0000 7f5ee9c1a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:31.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.784+0000 7f5ee9c1a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:31.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.784+0000 7f5ee9c1a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:31.787 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.784+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5ee414eb30 con 0x7f5ee40c93d0 2026-02-22T21:41:31.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.787+0000 7f5ebffff640 1 -- 10.20.193.51:0/3883043842 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f5ed40869d0 con 0x7f5ee40c93d0 2026-02-22T21:41:31.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.787+0000 7f5ebffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:31.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.787+0000 7f5ebffff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:31.791 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:31.791 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:31.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 10 librados: watch_flush enter 2026-02-22T21:41:31.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 10 librados: watch_flush exit 2026-02-22T21:41:31.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 20 client.5022.objecter shutdown clearing up homeless session... 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 10 client.5022.objecter successfully canceled tick 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 msgr2=0x7f5eb407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 0x7f5eb407b2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0002770 tx=0x7f5ed0007ab0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 10 monclient: shutdown 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 msgr2=0x7f5ee41f74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009aa0 tx=0x7f5ed40047c0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 shutdown_connections 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5eb4078e00 0x7f5eb407b2c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5ee4152ab0 0x7f5ee415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5ee40c93d0 0x7f5ee41f74d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 --2- 10.20.193.51:0/3883043842 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5ee406d9d0 0x7f5ee4073de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 >> 10.20.193.51:0/3883043842 conn(0x7f5ee40cd7b0 msgr2=0x7f5ee40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 shutdown_connections 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 -- 10.20.193.51:0/3883043842 wait complete. 2026-02-22T21:41:31.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:31.788+0000 7f5ee9c1a640 1 librados: shutdown 2026-02-22T21:41:34.800 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:34.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.865+0000 7f491a2cd640 10 monclient: get_monmap_and_config 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.865+0000 7f491a2cd640 10 monclient: build_initial_monmap 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.865+0000 7f491a2cd640 10 monclient: monmap: 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:34.867448+0000 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:34.867448+0000 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:34.869 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 Processor -- start 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 -- start start 2026-02-22T21:41:34.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient: init 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient: _add_conns ranks=[5,4,0,3,1,2] 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4914150220 0x7f49140c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4914150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f4914150600 0x7f491415c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4914150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f491415c540 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f491415c9b0 con 0x7f491415c540 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f491415a920 con 0x7f4914150600 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f491415af60 con 0x7f4914150220 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.866+0000 7f491a2cd640 10 monclient(hunting): authenticate will time out at 1370.447762s 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4918843640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f491415c540 0x7f491406fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58850/0 (socket says 10.20.193.51:58850) 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4918843640 1 -- 10.20.193.51:0/3086809513 learned_addr learned my addr 10.20.193.51:0/3086809513 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 1 --2- 10.20.193.51:0/3086809513 >> v2:10.20.193.68:3300/0 conn(0x7f4914150600 0x7f491415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 1 --2- 10.20.193.51:0/3086809513 >> v2:10.20.193.125:3300/0 conn(0x7f4914150220 0x7f49140c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): get_auth_request con 0x7f4914150600 auth_method 0 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): get_auth_request con 0x7f4914150220 auth_method 0 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:34.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.867+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 10 monclient(hunting): handle_auth_done global_id 5026 payload 274 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 1 -- 10.20.193.51:0/3086809513 >> v2:10.20.193.125:3300/0 conn(0x7f4914150220 msgr2=0x7f49140c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 1 --2- 10.20.193.51:0/3086809513 >> v2:10.20.193.125:3300/0 conn(0x7f4914150220 0x7f49140c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 1 -- 10.20.193.51:0/3086809513 >> v1:10.20.193.51:6789/0 conn(0x7f491415c540 legacy=0x7f491406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 1 monclient: found mon.noname-b 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.868+0000 7f49137fe640 1 -- 10.20.193.51:0/3086809513 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491415b240 con 0x7f4914150600 2026-02-22T21:41:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.869+0000 7f49137fe640 1 --2- 10.20.193.51:0/3086809513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150600 0x7f491415c000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4914070940 tx=0x7f490402eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f53db5c5790031 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 1 -- 10.20.193.51:0/3086809513 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490403c070 con 0x7f4914150600 2026-02-22T21:41:34.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: dump: 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:34.884 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: _finish_auth 0 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f491a2cd640 5 monclient: authenticate success, global_id 5026 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 1 -- 10.20.193.51:0/3086809513 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f490402fb80 con 0x7f4914150600 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 1 -- 10.20.193.51:0/3086809513 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904037440 con 0x7f4914150600 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f491a2cd640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f491a2cd640 4 set_mon_vals no callback set 2026-02-22T21:41:34.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.871+0000 7f4912ffd640 10 monclient: dump: 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:34.886 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 10 monclient: shutdown 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 -- 10.20.193.51:0/3086809513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150600 msgr2=0x7f491415c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 --2- 10.20.193.51:0/3086809513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150600 0x7f491415c000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4914070940 tx=0x7f490402eeb0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 -- 10.20.193.51:0/3086809513 shutdown_connections 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 --2- 10.20.193.51:0/3086809513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150600 0x7f491415c000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 --2- 10.20.193.51:0/3086809513 >> v2:10.20.193.125:3300/0 conn(0x7f4914150220 0x7f49140c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 -- 10.20.193.51:0/3086809513 >> 10.20.193.51:0/3086809513 conn(0x7f49140cd7b0 msgr2=0x7f49140cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 -- 10.20.193.51:0/3086809513 shutdown_connections 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 1 -- 10.20.193.51:0/3086809513 wait complete. 2026-02-22T21:41:34.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 10 monclient: build_initial_monmap 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.872+0000 7f491a2cd640 10 monclient: monmap: 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 librados: starting msgr at 2026-02-22T21:41:34.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 librados: starting objecter 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 Processor -- start 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 -- start start 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 librados: setting wanted keys 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 librados: calling monclient init 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient: init 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4914150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4914150600 0x7f49141f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4914150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:34.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f491415c540 0x7f49141e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): picked mon.noname-a con 0x7f491415c540 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f491415bae0 con 0x7f491415c540 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f491415b800 con 0x7f4914150220 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f49140703b0 con 0x7f4914150600 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:34.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f491a2cd640 10 monclient(hunting): authenticate will time out at 1370.454607s 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f4918843640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f491415c540 0x7f49141e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.873+0000 7f4913fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4914150600 0x7f49141f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47478/0 (socket says 10.20.193.51:47478) 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 1 -- 10.20.193.51:0/624872172 learned_addr learned my addr 10.20.193.51:0/624872172 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): get_auth_request con 0x7f4914150220 auth_method 0 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): get_auth_request con 0x7f491415c540 auth_method 0 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:34.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): get_auth_request con 0x7f4914150600 auth_method 0 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4913fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f4918843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:34.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.874+0000 7f49137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 10 monclient(hunting): handle_auth_done global_id 5029 payload 824 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 -- 10.20.193.51:0/624872172 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4914150600 msgr2=0x7f49141f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4914150600 0x7f49141f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 -- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f491415c540 msgr2=0x7f49141e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f491415c540 0x7f49141e7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 monclient: found mon.noname-b 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:34.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.875+0000 7f4913fff640 1 -- 10.20.193.51:0/624872172 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49140c9ff0 con 0x7f4914150220 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.876+0000 7f4913fff640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f490000ebe0 tx=0x7f490000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.876+0000 7f4910ff9640 10 client.?.objecter ms_handle_connect 0x7f4914150220 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.876+0000 7f4910ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900017070 con 0x7f4914150220 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: dump: 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:34.894 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: _finish_auth 0 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 5 monclient: authenticate success, global_id 5029 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f490000cc40 con 0x7f4914150220 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900013470 con 0x7f4914150220 2026-02-22T21:41:34.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f49127fc640 4 set_mon_vals no callback set 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f4910ff9640 10 monclient: dump: 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:34.896 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 10 monclient: _renew_subs 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49140ca2d0 con 0x7f4914150220 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 10 client.5029.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 10 monclient: _renew_subs 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49140ca890 con 0x7f4914150220 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.878+0000 7f491a2cd640 1 librados: init done 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.879+0000 7f491a2cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.879+0000 7f491a2cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.879+0000 7f491a2cd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:34.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.879+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f491415c000 con 0x7f4914150220 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.890+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4900013610 con 0x7f4914150220 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.890+0000 7f4910ff9640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 0x7f48e407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.890+0000 7f49137fe640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 0x7f48e407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.890+0000 7f49137fe640 10 monclient: get_auth_request con 0x7f48e4078e50 auth_method 0 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.891+0000 7f49137fe640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 0x7f48e407b310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4904039000 tx=0x7f4904033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:34.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.891+0000 7f4910ff9640 10 client.5029.objecter ms_handle_connect 0x7f48e4078e50 2026-02-22T21:41:34.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.892+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f490005e560 con 0x7f4914150220 2026-02-22T21:41:34.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.892+0000 7f4910ff9640 10 client.5029.objecter ms_dispatch2 0x7f4914000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:34.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.892+0000 7f4910ff9640 3 client.5029.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:34.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.892+0000 7f4910ff9640 3 client.5029.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:34.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.892+0000 7f4910ff9640 20 client.5029.objecter dump_active .. 0 homeless 2026-02-22T21:41:34.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.994+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f4900062520 con 0x7f4914150220 2026-02-22T21:41:34.996 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.994+0000 7f4910ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:34.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:34.994+0000 7f4910ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:35.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.092+0000 7f491a2cd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:35.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.092+0000 7f491a2cd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:35.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.092+0000 7f491a2cd640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:35.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.092+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4914074da0 con 0x7f4914150220 2026-02-22T21:41:35.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.095+0000 7f4910ff9640 1 -- 10.20.193.51:0/624872172 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f4900066820 con 0x7f4914150220 2026-02-22T21:41:35.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.095+0000 7f4910ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:35.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.095+0000 7f4910ff9640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:35.098 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:35.098 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:35.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 10 librados: watch_flush enter 2026-02-22T21:41:35.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 10 librados: watch_flush exit 2026-02-22T21:41:35.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 20 client.5029.objecter shutdown clearing up homeless session... 2026-02-22T21:41:35.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 10 client.5029.objecter successfully canceled tick 2026-02-22T21:41:35.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 msgr2=0x7f48e407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 0x7f48e407b310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4904039000 tx=0x7f4904033000 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 10 monclient: shutdown 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 msgr2=0x7f49140c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f490000ebe0 tx=0x7f490000c520 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 shutdown_connections 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f48e4078e50 0x7f48e407b310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f491415c540 0x7f49141e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4914150600 0x7f49141f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 --2- 10.20.193.51:0/624872172 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4914150220 0x7f49140c9990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:35.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.096+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 >> 10.20.193.51:0/624872172 conn(0x7f49140cd7b0 msgr2=0x7f491406eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:35.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.097+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 shutdown_connections 2026-02-22T21:41:35.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.097+0000 7f491a2cd640 1 -- 10.20.193.51:0/624872172 wait complete. 2026-02-22T21:41:35.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:35.097+0000 7f491a2cd640 1 librados: shutdown 2026-02-22T21:41:38.109 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:38.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.173+0000 7fcba5349640 10 monclient: get_monmap_and_config 2026-02-22T21:41:38.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.173+0000 7fcba5349640 10 monclient: build_initial_monmap 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.173+0000 7fcba5349640 10 monclient: monmap: 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:38.174357+0000 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:38.174357+0000 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:38.176 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 Processor -- start 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 -- start start 2026-02-22T21:41:38.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient: init 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient: _add_conns ranks=[2,5,3,0,1,4] 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcba0150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcba015c710 0x7fcba00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-c con 0x7fcba015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcba0150600 0x7fcba006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-a con 0x7fcba0150600 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba015a920 con 0x7fcba0150220 2026-02-22T21:41:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fcba015af60 con 0x7fcba0150600 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fcba015b240 con 0x7fcba015c710 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcba5349640 10 monclient(hunting): authenticate will time out at 1373.754649s 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.174+0000 7fcb9f7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcba0150600 0x7fcba006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcba0150600 0x7fcba006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43020/0 (socket says 10.20.193.51:43020) 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/862325855 learned_addr learned my addr 10.20.193.51:0/862325855 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcba0150600 auth_method 0 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/862325855 >> v2:10.20.193.125:3300/0 conn(0x7fcba015c710 0x7fcba00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcba015c710 auth_method 0 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.175+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_done global_id 5031 payload 274 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/862325855 >> v2:10.20.193.125:3300/0 conn(0x7fcba015c710 msgr2=0x7fcba00c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/862325855 >> v2:10.20.193.125:3300/0 conn(0x7fcba015c710 0x7fcba00c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/862325855 >> v1:10.20.193.125:6789/0 conn(0x7fcba0150220 legacy=0x7fcba015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.176+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/862325855 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba015b520 con 0x7fcba0150600 2026-02-22T21:41:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.177+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/862325855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba006fca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009e50 tx=0x7fcb9002f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d914a1404e857d2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.178+0000 7fcb9dffb640 1 -- 10.20.193.51:0/862325855 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9002fdf0 con 0x7fcba0150600 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: dump: 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:38.189 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: _finish_auth 0 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 1 -- 10.20.193.51:0/862325855 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb900373f0 con 0x7fcba0150600 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 5 monclient: authenticate success, global_id 5031 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 1 -- 10.20.193.51:0/862325855 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb900377c0 con 0x7fcba0150600 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcb9dffb640 10 monclient: dump: 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:38.191 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 4 set_mon_vals no callback set 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 10 monclient: shutdown 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 -- 10.20.193.51:0/862325855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 msgr2=0x7fcba006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 --2- 10.20.193.51:0/862325855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba006fca0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009e50 tx=0x7fcb9002f330 comp rx=0 tx=0).stop 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 -- 10.20.193.51:0/862325855 shutdown_connections 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 --2- 10.20.193.51:0/862325855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba006fca0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 --2- 10.20.193.51:0/862325855 >> v2:10.20.193.125:3300/0 conn(0x7fcba015c710 0x7fcba00c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 -- 10.20.193.51:0/862325855 >> 10.20.193.51:0/862325855 conn(0x7fcba00cd7b0 msgr2=0x7fcba00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 -- 10.20.193.51:0/862325855 shutdown_connections 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 1 -- 10.20.193.51:0/862325855 wait complete. 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 10 monclient: build_initial_monmap 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.179+0000 7fcba5349640 10 monclient: monmap: 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:38.193 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 librados: starting msgr at 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 librados: starting objecter 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 Processor -- start 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 -- start start 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 librados: setting wanted keys 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 librados: calling monclient init 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient: init 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcba014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-c con 0x7fcba0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-a con 0x7fcba0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcba006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): picked mon.noname-b con 0x7fcba015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcba00701e0 con 0x7fcba0150600 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcba015bae0 con 0x7fcba015c710 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcba0070360 con 0x7fcba0150220 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:38.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcba5349640 10 monclient(hunting): authenticate will time out at 1373.760533s 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43028/0 (socket says 10.20.193.51:43028) 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 learned_addr learned my addr 10.20.193.51:0/3474705682 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcba0150600 auth_method 0 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.180+0000 7fcb9effd640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcba014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcba006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): get_auth_request con 0x7fcba0150220 auth_method 0 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcba015c710 auth_method 0 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:38.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.181+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_done global_id 5034 payload 824 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 msgr2=0x7fcba014aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcba014aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 msgr2=0x7fcba006ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcba006ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:41:38.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 monclient: found mon.noname-a 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.182+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba01e7c80 con 0x7fcba0150600 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.183+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e890 tx=0x7fcb8800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.183+0000 7fcb7f7fe640 10 client.?.objecter ms_handle_connect 0x7fcba0150600 2026-02-22T21:41:38.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.183+0000 7fcb7f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88017070 con 0x7fcba0150600 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: dump: 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.200 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb8800c940 con 0x7fcba0150600 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8800cd30 con 0x7fcba0150600 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb9d7fa640 4 set_mon_vals no callback set 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcb7f7fe640 10 monclient: dump: 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcba5349640 5 monclient: authenticate success, global_id 5034 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcba5349640 10 monclient: _renew_subs 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcba5349640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.184+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba006d000 con 0x7fcba0150600 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 client.5034.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 monclient: _renew_subs 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba006d5c0 con 0x7fcba0150600 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 1 librados: init done 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:38.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.185+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba006fca0 con 0x7fcba0150600 2026-02-22T21:41:38.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.289+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcb88013470 con 0x7fcba0150600 2026-02-22T21:41:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.290+0000 7fcb7f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 0x7fcb6c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.290+0000 7fcb9effd640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 0x7fcb6c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.290+0000 7fcb9effd640 10 monclient: get_auth_request con 0x7fcb6c078e00 auth_method 0 2026-02-22T21:41:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.290+0000 7fcb7f7fe640 10 client.5034.objecter ms_handle_connect 0x7fcb6c078e00 2026-02-22T21:41:38.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.290+0000 7fcb9effd640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 0x7fcb6c07b2c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcb940051f0 tx=0x7fcb94007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fcb8805e3e0 con 0x7fcba0150600 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 msgr2=0x7fcba01e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 client.5034.objecter ms_dispatch2 0x7fcba0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb9e7fc640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 msgr2=0x7fcba01e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 3 client.5034.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 3 client.5034.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e890 tx=0x7fcb8800ed60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e890 tx=0x7fcb8800ed60 comp rx=0 tx=0).stop 2026-02-22T21:41:38.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 20 client.5034.objecter dump_active .. 0 homeless 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient: ms_handle_reset current mon [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 msgr2=0x7fcba01e8ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcba0150600 0x7fcba01e8ae0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcb6c082cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): picked mon.b con 0x7fcba015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcb6c085500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): picked mon.c con 0x7fcba0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcb6c085ed0 0x7fcb6c088360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): picked mon.a con 0x7fcb6c085ed0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c088dd0 con 0x7fcb6c085ed0 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c088f80 con 0x7fcba015c710 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c089180 con 0x7fcba0150220 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.298+0000 7fcb7f7fe640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcb6c082cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcb6c085500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:38.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request con 0x7fcba0150220 auth_method 0 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request con 0x7fcba015c710 auth_method 0 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:38.306 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.299+0000 7fcb9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient(hunting): handle_auth_done global_id 5034 payload 293 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 msgr2=0x7fcb6c082cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcb6c082cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcb6c085ed0 msgr2=0x7fcb6c088360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcb6c085ed0 0x7fcb6c088360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 monclient: found mon.c 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fcb6c0893e0 con 0x7fcba0150220 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:38.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba015b520 con 0x7fcba0150220 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.300+0000 7fcb9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.301+0000 7fcb9f7fe640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcb6c085500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb90037660 tx=0x7fcb88009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.301+0000 7fcb7f7fe640 10 client.5034.objecter ms_handle_connect 0x7fcba0150220 2026-02-22T21:41:38.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.301+0000 7fcb7f7fe640 10 client.5034.objecter resend_mon_ops 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.302+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90046070 con 0x7fcba0150220 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.302+0000 7fcb7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.302+0000 7fcb7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.302+0000 7fcb7f7fe640 10 monclient: dump: 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:38.309 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:38.310 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:38.310 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:38.310 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:38.310 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:38.310 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.406+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb90037710 con 0x7fcba0150220 2026-02-22T21:41:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.406+0000 7fcb7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.406+0000 7fcb9cff9640 4 set_mon_vals no callback set 2026-02-22T21:41:38.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.407+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcb900378c0 con 0x7fcba0150220 2026-02-22T21:41:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.414+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fcb900854a0 con 0x7fcba0150220 2026-02-22T21:41:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.414+0000 7fcb7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:38.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.414+0000 7fcb7f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:38.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.517+0000 7fcba5349640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.517+0000 7fcba5349640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.517+0000 7fcba5349640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:38.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.517+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcba0074da0 con 0x7fcba0150220 2026-02-22T21:41:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.519+0000 7fcb7f7fe640 1 -- 10.20.193.51:0/3474705682 <== mon.2 v2:10.20.193.125:3300/0 5 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fcb900897a0 con 0x7fcba0150220 2026-02-22T21:41:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.519+0000 7fcb7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:38.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.519+0000 7fcb7f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:38.521 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:38.522 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 10 librados: watch_flush enter 2026-02-22T21:41:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 10 librados: watch_flush exit 2026-02-22T21:41:38.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 20 client.5034.objecter shutdown clearing up homeless session... 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 10 client.5034.objecter successfully canceled tick 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 msgr2=0x7fcb6c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 0x7fcb6c07b2c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcb940051f0 tx=0x7fcb94007f30 comp rx=0 tx=0).stop 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 10 monclient: shutdown 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 msgr2=0x7fcb6c085500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcb6c085500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcb90037660 tx=0x7fcb88009290 comp rx=0 tx=0).stop 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 shutdown_connections 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcb6c085ed0 0x7fcb6c088360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcba0150220 0x7fcb6c085500 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcba015c710 0x7fcb6c082cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 --2- 10.20.193.51:0/3474705682 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcb6c078e00 0x7fcb6c07b2c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 >> 10.20.193.51:0/3474705682 conn(0x7fcba00cd7b0 msgr2=0x7fcba014cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 shutdown_connections 2026-02-22T21:41:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.520+0000 7fcba5349640 1 -- 10.20.193.51:0/3474705682 wait complete. 2026-02-22T21:41:38.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:38.521+0000 7fcba5349640 1 librados: shutdown 2026-02-22T21:41:41.532 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:41.582 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:12.735 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 10 monclient: get_monmap_and_config 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 10 monclient: build_initial_monmap 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 10 monclient: monmap: 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:41.600747+0000 2026-02-22T21:41:41.601 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:41.600747+0000 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:41.602 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 1 Processor -- start 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 1 -- start start 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.600+0000 7f0b6143c640 10 monclient: init 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient: _add_conns ranks=[3,2,4,1,5,0] 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0b5c150220 0x7f0b5c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b5c150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f0b5c15c710 0x7f0b5c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c15c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b5c15a920 con 0x7f0b5c150600 2026-02-22T21:41:41.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f0b5c15af60 con 0x7f0b5c150220 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f0b5c15b240 con 0x7f0b5c15c710 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b6143c640 10 monclient(hunting): authenticate will time out at 1377.181130s 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b5a7fc640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f0b5c150600 0x7f0b5c15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:53922/0 (socket says 10.20.193.51:53922) 2026-02-22T21:41:41.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.601+0000 7f0b5a7fc640 1 -- 10.20.193.51:0/1763628895 learned_addr learned my addr 10.20.193.51:0/1763628895 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:41.610 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.609+0000 7f0b5affd640 1 --2- 10.20.193.51:0/1763628895 >> v2:10.20.193.51:3300/0 conn(0x7f0b5c150220 0x7f0b5c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:41.611 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:12.762 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.609+0000 7f0b5affd640 10 monclient(hunting): get_auth_request con 0x7f0b5c150220 auth_method 0 2026-02-22T21:41:41.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.609+0000 7f0b5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:41.611 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.609+0000 7f0b5affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:41.612 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.609+0000 7f0b5affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:41.612 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:12.765 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.611+0000 7f0b5b7fe640 1 --2- 10.20.193.51:0/1763628895 >> v2:10.20.193.68:3300/0 conn(0x7f0b5c15c710 0x7f0b5c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.612+0000 7f0b5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0b5c15c710 auth_method 0 2026-02-22T21:41:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.612+0000 7f0b5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:41.613 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.612+0000 7f0b5b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:41.614 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.612+0000 7f0b5b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:41.614 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:12.765 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.613+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.613+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:41.615 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.613+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:41.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 10 monclient(hunting): handle_auth_done global_id 5037 payload 274 2026-02-22T21:41:41.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 1 -- 10.20.193.51:0/1763628895 >> v2:10.20.193.68:3300/0 conn(0x7f0b5c15c710 msgr2=0x7f0b5c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:41.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 1 --2- 10.20.193.51:0/1763628895 >> v2:10.20.193.68:3300/0 conn(0x7f0b5c15c710 0x7f0b5c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 1 -- 10.20.193.51:0/1763628895 >> v1:10.20.193.125:6789/0 conn(0x7f0b5c150600 legacy=0x7f0b5c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.616 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:41.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 1 monclient: found mon.noname-a 2026-02-22T21:41:41.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:41.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:41.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.614+0000 7f0b5affd640 1 -- 10.20.193.51:0/1763628895 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5c15b520 con 0x7f0b5c150220 2026-02-22T21:41:41.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.616+0000 7f0b5affd640 1 --2- 10.20.193.51:0/1763628895 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150220 0x7f0b5c0c73a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0c65c0 tx=0x7f0b5002eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5c5869bdf02c69 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:41.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 1 -- 10.20.193.51:0/1763628895 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5003c070 con 0x7f0b5c150220 2026-02-22T21:41:41.624 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: dump: 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:41.625 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: _finish_auth 0 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 1 -- 10.20.193.51:0/1763628895 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5002fb80 con 0x7f0b5c150220 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 5 monclient: authenticate success, global_id 5037 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 1 -- 10.20.193.51:0/1763628895 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50037440 con 0x7f0b5c150220 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:41.626 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b59ffb640 10 monclient: dump: 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.627 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 4 set_mon_vals no callback set 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 10 monclient: shutdown 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 -- 10.20.193.51:0/1763628895 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150220 msgr2=0x7f0b5c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 --2- 10.20.193.51:0/1763628895 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150220 0x7f0b5c0c73a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0c65c0 tx=0x7f0b5002eeb0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 -- 10.20.193.51:0/1763628895 shutdown_connections 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 --2- 10.20.193.51:0/1763628895 >> v2:10.20.193.68:3300/0 conn(0x7f0b5c15c710 0x7f0b5c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 --2- 10.20.193.51:0/1763628895 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150220 0x7f0b5c0c73a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 -- 10.20.193.51:0/1763628895 >> 10.20.193.51:0/1763628895 conn(0x7f0b5c0cd7b0 msgr2=0x7f0b5c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 -- 10.20.193.51:0/1763628895 shutdown_connections 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.620+0000 7f0b6143c640 1 -- 10.20.193.51:0/1763628895 wait complete. 2026-02-22T21:41:41.628 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient: build_initial_monmap 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient: monmap: 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:41.629 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 librados: starting msgr at 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 librados: starting objecter 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 Processor -- start 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 -- start start 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 librados: setting wanted keys 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 librados: calling monclient init 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient: init 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:41.630 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150600 0x7f0b5c1e9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0b5c15c710 0x7f0b5c1ea430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c0701e0 con 0x7f0b5c150600 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c15bae0 con 0x7f0b5c150220 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5c070360 con 0x7f0b5c15c710 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.621+0000 7f0b6143c640 10 monclient(hunting): authenticate will time out at 1377.201870s 2026-02-22T21:41:41.631 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.622+0000 7f0b5affd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47264/0 (socket says 10.20.193.51:47264) 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 1 -- 10.20.193.51:0/4139778924 learned_addr learned my addr 10.20.193.51:0/4139778924 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): get_auth_request con 0x7f0b5c150220 auth_method 0 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:41.632 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:41.633 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:12.780 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.623+0000 7f0b5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 10 monclient(hunting): handle_auth_done global_id 5038 payload 824 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 -- 10.20.193.51:0/4139778924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0b5c15c710 msgr2=0x7f0b5c1ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0b5c15c710 0x7f0b5c1ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 -- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150600 msgr2=0x7f0b5c1e9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.634 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150600 0x7f0b5c1e9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 monclient: found mon.noname-b 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.625+0000 7f0b5affd640 1 -- 10.20.193.51:0/4139778924 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5c1e9a40 con 0x7f0b5c150220 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.628+0000 7f0b5affd640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009130 tx=0x7f0b500047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.628+0000 7f0b3b7fe640 10 client.?.objecter ms_handle_connect 0x7f0b5c150220 2026-02-22T21:41:41.635 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.628+0000 7f0b3b7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:41.636 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:12.785 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5003c040 con 0x7f0b5c150220 2026-02-22T21:41:41.637 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: dump: 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:41.638 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b50042450 con 0x7f0b5c150220 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 5 monclient: authenticate success, global_id 5038 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 10 monclient: _renew_subs 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b5c06d090 con 0x7f0b5c150220 2026-02-22T21:41:41.639 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50042820 con 0x7f0b5c150220 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b3b7fe640 10 monclient: dump: 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:41.640 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b597fa640 4 set_mon_vals no callback set 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 10 client.5038.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 10 monclient: _renew_subs 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b5c06d650 con 0x7f0b5c150220 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.633+0000 7f0b6143c640 1 librados: init done 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.634+0000 7f0b6143c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.634+0000 7f0b6143c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.634+0000 7f0b6143c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:41.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.634+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b5c06fca0 con 0x7f0b5c150220 2026-02-22T21:41:41.642 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:12.757 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.643 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:12.759 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.650 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:12.766 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:41:41.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.745+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f0b500429c0 con 0x7f0b5c150220 2026-02-22T21:41:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.746+0000 7f0b3b7fe640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 0x7f0b2807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.746+0000 7f0b5a7fc640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 0x7f0b2807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.746+0000 7f0b5a7fc640 10 monclient: get_auth_request con 0x7f0b28078e50 auth_method 0 2026-02-22T21:41:41.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.746+0000 7f0b3b7fe640 10 client.5038.objecter ms_handle_connect 0x7f0b28078e50 2026-02-22T21:41:41.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.746+0000 7f0b5a7fc640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 0x7f0b2807b310 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0b44007fd0 tx=0x7f0b44007b40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:41.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.747+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f0b500837a0 con 0x7f0b5c150220 2026-02-22T21:41:41.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.747+0000 7f0b3b7fe640 10 client.5038.objecter ms_dispatch2 0x7f0b5c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.747+0000 7f0b3b7fe640 3 client.5038.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.747+0000 7f0b3b7fe640 3 client.5038.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:41.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.747+0000 7f0b3b7fe640 20 client.5038.objecter dump_active .. 0 homeless 2026-02-22T21:41:41.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.754+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f0b50087e60 con 0x7f0b5c150220 2026-02-22T21:41:41.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.754+0000 7f0b3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:41.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.754+0000 7f0b3b7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.854+0000 7f0b6143c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.854+0000 7f0b6143c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.854+0000 7f0b6143c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:41.856 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.854+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0b5c074da0 con 0x7f0b5c150220 2026-02-22T21:41:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.857+0000 7f0b3b7fe640 1 -- 10.20.193.51:0/4139778924 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f0b50087760 con 0x7f0b5c150220 2026-02-22T21:41:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.857+0000 7f0b3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:41.859 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.857+0000 7f0b3b7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:41.859 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:41.859 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 10 librados: watch_flush enter 2026-02-22T21:41:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 10 librados: watch_flush exit 2026-02-22T21:41:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 20 client.5038.objecter shutdown clearing up homeless session... 2026-02-22T21:41:41.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 10 client.5038.objecter successfully canceled tick 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 msgr2=0x7f0b2807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 0x7f0b2807b310 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0b44007fd0 tx=0x7f0b44007b40 comp rx=0 tx=0).stop 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 10 monclient: shutdown 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 msgr2=0x7f0b5c0737f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009130 tx=0x7f0b500047c0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 shutdown_connections 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0b28078e50 0x7f0b2807b310 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0b5c15c710 0x7f0b5c1ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0b5c150600 0x7f0b5c1e9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 --2- 10.20.193.51:0/4139778924 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0b5c150220 0x7f0b5c0737f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 >> 10.20.193.51:0/4139778924 conn(0x7f0b5c0cd7b0 msgr2=0x7f0b5c0c01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 shutdown_connections 2026-02-22T21:41:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 -- 10.20.193.51:0/4139778924 wait complete. 2026-02-22T21:41:41.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:41.858+0000 7f0b6143c640 1 librados: shutdown 2026-02-22T21:41:44.868 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:44.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.934+0000 7fe35e21a640 10 monclient: get_monmap_and_config 2026-02-22T21:41:44.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.934+0000 7fe35e21a640 10 monclient: build_initial_monmap 2026-02-22T21:41:44.936 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.934+0000 7fe35e21a640 10 monclient: monmap: 2026-02-22T21:41:44.936 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:44.936 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:44.935400+0000 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:44.935400+0000 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:44.937 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 Processor -- start 2026-02-22T21:41:44.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 -- start start 2026-02-22T21:41:44.940 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient: init 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient: _add_conns ranks=[0,1,4,5,2,3] 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe35815f480 addr v1:10.20.193.51:6789/0 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe35815d110 addr v1:10.20.193.68:6789/0 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fe3580713b0 0x7fe35815f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-b con 0x7fe3580713b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3580c8620 con 0x7fe35815f480 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3580c8900 con 0x7fe35815d110 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fe3580c9510 con 0x7fe3580713b0 2026-02-22T21:41:44.941 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.935+0000 7fe35e21a640 10 monclient(hunting): authenticate will time out at 1380.515801s 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe3577fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fe35815f480 0x7fe3580c99a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:56020/0 (socket says 10.20.193.51:56020) 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe3577fe640 1 -- 10.20.193.51:0/4055666902 learned_addr learned my addr 10.20.193.51:0/4055666902 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 1 --2- 10.20.193.51:0/4055666902 >> v2:10.20.193.68:3300/0 conn(0x7fe3580713b0 0x7fe35815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 10 monclient(hunting): get_auth_request con 0x7fe3580713b0 auth_method 0 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.936+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:44.942 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 1 -- 10.20.193.51:0/4055666902 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364893440 0 0) 0x7fe3580c8620 con 0x7fe35815f480 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 10 monclient(hunting): _negotiate 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 10 monclient(hunting): my global_id is 5040 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.937+0000 7fe3567fc640 1 -- 10.20.193.51:0/4055666902 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe33c003620 con 0x7fe35815f480 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 10 monclient(hunting): handle_auth_done global_id 5041 payload 274 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 1 -- 10.20.193.51:0/4055666902 >> v1:10.20.193.68:6789/0 conn(0x7fe35815d110 legacy=0x7fe3580736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 1 -- 10.20.193.51:0/4055666902 >> v1:10.20.193.51:6789/0 conn(0x7fe35815f480 legacy=0x7fe3580c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 1 monclient: found mon.noname-b 2026-02-22T21:41:44.943 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:44.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:44.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.938+0000 7fe357fff640 1 -- 10.20.193.51:0/4055666902 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3580c8be0 con 0x7fe3580713b0 2026-02-22T21:41:44.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.940+0000 7fe357fff640 1 --2- 10.20.193.51:0/4055666902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe3580713b0 0x7fe35815f860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009d90 tx=0x7fe34c02f230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4a1b06bb407add1 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 1 -- 10.20.193.51:0/4055666902 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c03c070 con 0x7fe3580713b0 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: dump: 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.952 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: _finish_auth 0 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: _check_auth_tickets 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 1 -- 10.20.193.51:0/4055666902 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe34c037440 con 0x7fe3580713b0 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe35e21a640 5 monclient: authenticate success, global_id 5041 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 1 -- 10.20.193.51:0/4055666902 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c037810 con 0x7fe3580713b0 2026-02-22T21:41:44.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe3567fc640 10 monclient: dump: 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:44.954 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe35e21a640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.941+0000 7fe35e21a640 4 set_mon_vals no callback set 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 10 monclient: shutdown 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- 10.20.193.51:0/4055666902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe3580713b0 msgr2=0x7fe35815f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 --2- 10.20.193.51:0/4055666902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe3580713b0 0x7fe35815f860 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009d90 tx=0x7fe34c02f230 comp rx=0 tx=0).stop 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- 10.20.193.51:0/4055666902 shutdown_connections 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 --2- 10.20.193.51:0/4055666902 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe3580713b0 0x7fe35815f860 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- 10.20.193.51:0/4055666902 >> 10.20.193.51:0/4055666902 conn(0x7fe3580ca680 msgr2=0x7fe3580caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:44.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- 10.20.193.51:0/4055666902 shutdown_connections 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- 10.20.193.51:0/4055666902 wait complete. 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 10 monclient: build_initial_monmap 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 10 monclient: monmap: 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:44.956 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 librados: starting msgr at 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 librados: starting objecter 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 Processor -- start 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 -- start start 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 librados: setting wanted keys 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 1 librados: calling monclient init 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.942+0000 7fe35e21a640 10 monclient: init 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 0x7fe358151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:44.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-a con 0x7fe3580713b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-b con 0x7fe35815d110 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe35815f480 0x7fe35814a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): picked mon.noname-c con 0x7fe35815f480 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe358076250 con 0x7fe3580713b0 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe3580c91a0 con 0x7fe35815d110 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe3580763d0 con 0x7fe35815f480 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:44.958 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe35e21a640 10 monclient(hunting): authenticate will time out at 1380.523340s 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 0x7fe358151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 0x7fe358151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:43054/0 (socket says 10.20.193.51:43054) 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 1 -- 10.20.193.51:0/384444597 learned_addr learned my addr 10.20.193.51:0/384444597 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47288/0 (socket says 10.20.193.51:47288) 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 10 monclient(hunting): get_auth_request con 0x7fe3580713b0 auth_method 0 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe3577fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 10 monclient(hunting): get_auth_request con 0x7fe35815d110 auth_method 0 2026-02-22T21:41:44.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe356ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.943+0000 7fe357fff640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe35815f480 0x7fe35814a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe356ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): get_auth_request con 0x7fe35815f480 auth_method 0 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:44.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe3577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.944+0000 7fe357fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 10 monclient(hunting): handle_auth_done global_id 5047 payload 824 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 -- 10.20.193.51:0/384444597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe35815f480 msgr2=0x7fe35814a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe35815f480 0x7fe35814a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 -- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 msgr2=0x7fe358151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 0x7fe358151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 monclient: found mon.noname-b 2026-02-22T21:41:44.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.945+0000 7fe356ffd640 1 -- 10.20.193.51:0/384444597 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3581505d0 con 0x7fe35815d110 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.946+0000 7fe356ffd640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c4e0 tx=0x7fe34800c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.946+0000 7fe333fff640 10 client.?.objecter ms_handle_connect 0x7fe35815d110 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.946+0000 7fe333fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3480052d0 con 0x7fe35815d110 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: dump: 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:44.962 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: _finish_auth 0 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:44.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe348005470 con 0x7fe35815d110 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 5 monclient: authenticate success, global_id 5047 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 10 monclient: _renew_subs 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3581508b0 con 0x7fe35815d110 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348003910 con 0x7fe35815d110 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe355ffb640 4 set_mon_vals no callback set 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe333fff640 10 monclient: dump: 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:44.964 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 10 client.5047.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 10 monclient: _renew_subs 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:44.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe358150e70 con 0x7fe35815d110 2026-02-22T21:41:44.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.948+0000 7fe35e21a640 1 librados: init done 2026-02-22T21:41:44.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.949+0000 7fe35e21a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:44.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.949+0000 7fe35e21a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:44.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.949+0000 7fe35e21a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:44.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:44.949+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe358076800 con 0x7fe35815d110 2026-02-22T21:41:45.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.053+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe348003ab0 con 0x7fe35815d110 2026-02-22T21:41:45.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.054+0000 7fe333fff640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 0x7fe32807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:45.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.054+0000 7fe3577fe640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 0x7fe32807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:45.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.054+0000 7fe3577fe640 10 monclient: get_auth_request con 0x7fe328078e00 auth_method 0 2026-02-22T21:41:45.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.054+0000 7fe3577fe640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 0x7fe32807b2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe344005980 tx=0x7fe344009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:45.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.054+0000 7fe333fff640 10 client.5047.objecter ms_handle_connect 0x7fe328078e00 2026-02-22T21:41:45.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.055+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe348097770 con 0x7fe35815d110 2026-02-22T21:41:45.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.055+0000 7fe333fff640 10 client.5047.objecter ms_dispatch2 0x7fe358000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:45.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.055+0000 7fe333fff640 3 client.5047.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:45.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.055+0000 7fe333fff640 3 client.5047.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:45.056 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.055+0000 7fe333fff640 20 client.5047.objecter dump_active .. 0 homeless 2026-02-22T21:41:45.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.061+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe348061f40 con 0x7fe35815d110 2026-02-22T21:41:45.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.061+0000 7fe333fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:45.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.061+0000 7fe333fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.166+0000 7fe35e21a640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.166+0000 7fe35e21a640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.166+0000 7fe35e21a640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:41:45.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.166+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe35806fb70 con 0x7fe35815d110 2026-02-22T21:41:45.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.168+0000 7fe333fff640 1 -- 10.20.193.51:0/384444597 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fe348066240 con 0x7fe35815d110 2026-02-22T21:41:45.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe333fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:45.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe333fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:45.171 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:45.171 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:45.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 10 librados: watch_flush enter 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 10 librados: watch_flush exit 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 20 client.5047.objecter shutdown clearing up homeless session... 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 10 client.5047.objecter successfully canceled tick 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 msgr2=0x7fe32807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 0x7fe32807b2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe344005980 tx=0x7fe344009040 comp rx=0 tx=0).stop 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 10 monclient: shutdown 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 msgr2=0x7fe358149ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.169+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c4e0 tx=0x7fe34800c9b0 comp rx=0 tx=0).stop 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 shutdown_connections 2026-02-22T21:41:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe328078e00 0x7fe32807b2c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe35815f480 0x7fe35814a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe35815d110 0x7fe358149ff0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 --2- 10.20.193.51:0/384444597 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe3580713b0 0x7fe358151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 >> 10.20.193.51:0/384444597 conn(0x7fe3580ca680 msgr2=0x7fe35806b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 shutdown_connections 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 -- 10.20.193.51:0/384444597 wait complete. 2026-02-22T21:41:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:45.170+0000 7fe35e21a640 1 librados: shutdown 2026-02-22T21:41:48.180 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 10 monclient: get_monmap_and_config 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 10 monclient: build_initial_monmap 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 10 monclient: monmap: 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:48.249 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:48.248000+0000 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:48.248000+0000 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 1 Processor -- start 2026-02-22T21:41:48.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 1 -- start start 2026-02-22T21:41:48.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.247+0000 7ff50187c640 10 monclient: init 2026-02-22T21:41:48.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:48.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient: _add_conns ranks=[2,4,3,1,0,5] 2026-02-22T21:41:48.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4fc06ff10 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ff4fc15a800 0x7ff4fc0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4fc15a800 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff4fc153750 0x7ff4fc153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4fc153750 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4fc15d150 con 0x7ff4fc06ff10 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff4fc15d3f0 con 0x7ff4fc153750 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7ff4fc15d6d0 con 0x7ff4fc15a800 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff50187c640 10 monclient(hunting): authenticate will time out at 1383.828302s 2026-02-22T21:41:48.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff4fc153750 0x7ff4fc153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fa7fc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7ff4fc15a800 0x7ff4fc0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff4fc153750 0x7ff4fc153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:49434/0 (socket says 10.20.193.51:49434) 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 1 -- 10.20.193.51:0/3357413905 learned_addr learned my addr 10.20.193.51:0/3357413905 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff4fc153750 auth_method 0 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fa7fc640 10 monclient(hunting): get_auth_request con 0x7ff4fc15a800 auth_method 0 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fa7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.248+0000 7ff4fa7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:48.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.249+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_done global_id 5046 payload 274 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 -- 10.20.193.51:0/3357413905 >> v2:10.20.193.68:3300/0 conn(0x7ff4fc15a800 msgr2=0x7ff4fc0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 --2- 10.20.193.51:0/3357413905 >> v2:10.20.193.68:3300/0 conn(0x7ff4fc15a800 0x7ff4fc0702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 -- 10.20.193.51:0/3357413905 >> v1:10.20.193.125:6789/0 conn(0x7ff4fc06ff10 legacy=0x7ff4fc15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 monclient: found mon.noname-a 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:48.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 -- 10.20.193.51:0/3357413905 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4fc15d9b0 con 0x7ff4fc153750 2026-02-22T21:41:48.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.250+0000 7ff4fb7fe640 1 --2- 10.20.193.51:0/3357413905 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc153750 0x7ff4fc153bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0c57d0 tx=0x7ff4ec02f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fad3c536e0513f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 1 -- 10.20.193.51:0/3357413905 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec02fdf0 con 0x7ff4fc153750 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: dump: 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:48.264 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: _finish_auth 0 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff50187c640 5 monclient: authenticate success, global_id 5046 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff50187c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 1 -- 10.20.193.51:0/3357413905 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ec0373f0 con 0x7ff4fc153750 2026-02-22T21:41:48.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 1 -- 10.20.193.51:0/3357413905 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec0377c0 con 0x7ff4fc153750 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff50187c640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff50187c640 4 set_mon_vals no callback set 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.252+0000 7ff4f9ffb640 10 monclient: dump: 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:48.266 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 10 monclient: shutdown 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 -- 10.20.193.51:0/3357413905 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc153750 msgr2=0x7ff4fc153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 --2- 10.20.193.51:0/3357413905 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc153750 0x7ff4fc153bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0c57d0 tx=0x7ff4ec02f2f0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 -- 10.20.193.51:0/3357413905 shutdown_connections 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 --2- 10.20.193.51:0/3357413905 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc153750 0x7ff4fc153bc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 --2- 10.20.193.51:0/3357413905 >> v2:10.20.193.68:3300/0 conn(0x7ff4fc15a800 0x7ff4fc0702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 -- 10.20.193.51:0/3357413905 >> 10.20.193.51:0/3357413905 conn(0x7ff4fc0cd7b0 msgr2=0x7ff4fc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 -- 10.20.193.51:0/3357413905 shutdown_connections 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 1 -- 10.20.193.51:0/3357413905 wait complete. 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 10 monclient: build_initial_monmap 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.253+0000 7ff50187c640 10 monclient: monmap: 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:48.268 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 librados: starting msgr at 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 librados: starting objecter 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 Processor -- start 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 -- start start 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 librados: setting wanted keys 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 librados: calling monclient init 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient: init 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4fc06ff10 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 0x7ff4fc15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4fc153750 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4fc15a800 0x7ff4fc15c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4fc15a800 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc159bd0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc15df70 con 0x7ff4fc153750 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff4fc0c9180 con 0x7ff4fc15a800 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:48.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff50187c640 10 monclient(hunting): authenticate will time out at 1383.834650s 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:49438/0 (socket says 10.20.193.51:49438) 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 1 -- 10.20.193.51:0/2400781783 learned_addr learned my addr 10.20.193.51:0/2400781783 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 10 monclient(hunting): get_auth_request con 0x7ff4fc06ff10 auth_method 0 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.254+0000 7ff4faffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 0x7ff4fc15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4fc15a800 0x7ff4fc15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): get_auth_request con 0x7ff4fc153750 auth_method 0 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:48.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): get_auth_request con 0x7ff4fc15a800 auth_method 0 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:48.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.255+0000 7ff4fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 10 monclient(hunting): handle_auth_done global_id 5049 payload 824 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 -- 10.20.193.51:0/2400781783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4fc15a800 msgr2=0x7ff4fc15c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4fc15a800 0x7ff4fc15c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 -- 10.20.193.51:0/2400781783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 msgr2=0x7ff4fc15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 0x7ff4fc15b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 10 monclient: _finish_hunting 0 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 monclient: found mon.noname-a 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4faffd640 1 -- 10.20.193.51:0/2400781783 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4fc073ed0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.256+0000 7ff4fa7fc640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 0x7ff4fc15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.259+0000 7ff4faffd640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000c5b0 tx=0x7ff4f000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.259+0000 7ff4d3fff640 10 client.?.objecter ms_handle_connect 0x7ff4fc06ff10 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.259+0000 7ff4d3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00052d0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: dump: 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:48.274 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: _finish_auth 0 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f0005470 con 0x7ff4fc06ff10 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00038f0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:48.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4f97fa640 4 set_mon_vals no callback set 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff4d3fff640 10 monclient: dump: 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:48.276 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 5 monclient: authenticate success, global_id 5049 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _renew_subs 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4fc0741b0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 client.5049.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _renew_subs 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4fc0bec20 con 0x7ff4fc06ff10 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 1 librados: init done 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.261+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4fc0702f0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.273+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff4f0003a90 con 0x7ff4fc06ff10 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.273+0000 7ff4d3fff640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 0x7ff4cc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.273+0000 7ff4fa7fc640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 0x7ff4cc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4fa7fc640 10 monclient: get_auth_request con 0x7ff4cc078e00 auth_method 0 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 10 client.5049.objecter ms_handle_connect 0x7ff4cc078e00 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4fa7fc640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 0x7ff4cc07b2c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009f90 tx=0x7ff4e4009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff4f005dfc0 con 0x7ff4fc06ff10 2026-02-22T21:41:48.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 10 client.5049.objecter ms_dispatch2 0x7ff4fc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:48.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 3 client.5049.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:48.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 3 client.5049.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:48.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.274+0000 7ff4d3fff640 20 client.5049.objecter dump_active .. 0 homeless 2026-02-22T21:41:48.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.374+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff4f0061f80 con 0x7ff4fc06ff10 2026-02-22T21:41:48.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.374+0000 7ff4d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:48.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.374+0000 7ff4d3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:48.474 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.472+0000 7ff50187c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.472+0000 7ff50187c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.472+0000 7ff50187c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:48.475 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.472+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff4fc157580 con 0x7ff4fc06ff10 2026-02-22T21:41:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.476+0000 7ff4d3fff640 1 -- 10.20.193.51:0/2400781783 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff4f0066280 con 0x7ff4fc06ff10 2026-02-22T21:41:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.476+0000 7ff4d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:48.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.476+0000 7ff4d3fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:48.478 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:48.478 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:48.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 10 librados: watch_flush enter 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 10 librados: watch_flush exit 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 20 client.5049.objecter shutdown clearing up homeless session... 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 10 client.5049.objecter successfully canceled tick 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 msgr2=0x7ff4cc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 0x7ff4cc07b2c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009f90 tx=0x7ff4e4009290 comp rx=0 tx=0).stop 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 10 monclient: shutdown 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 msgr2=0x7ff4fc0bf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff4f000c5b0 tx=0x7ff4f000ca80 comp rx=0 tx=0).stop 2026-02-22T21:41:48.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 shutdown_connections 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff4cc078e00 0x7ff4cc07b2c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff4fc15a800 0x7ff4fc15c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff4fc153750 0x7ff4fc15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 --2- 10.20.193.51:0/2400781783 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff4fc06ff10 0x7ff4fc0bf420 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 >> 10.20.193.51:0/2400781783 conn(0x7ff4fc0cd7b0 msgr2=0x7ff4fc0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 shutdown_connections 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 -- 10.20.193.51:0/2400781783 wait complete. 2026-02-22T21:41:48.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:48.477+0000 7ff50187c640 1 librados: shutdown 2026-02-22T21:41:51.489 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:51.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 10 monclient: get_monmap_and_config 2026-02-22T21:41:51.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 10 monclient: build_initial_monmap 2026-02-22T21:41:51.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 10 monclient: monmap: 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:51.557157+0000 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:51.557157+0000 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:51.559 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 1 Processor -- start 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 1 -- start start 2026-02-22T21:41:51.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.556+0000 7fba7b4bd640 10 monclient: init 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient: _add_conns ranks=[4,2,3,1,0,5] 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fba74150220 0x7fba740c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-b con 0x7fba74150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba74150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fba7415c710 0x7fba7406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-a con 0x7fba7415c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba7415a920 con 0x7fba74150600 2026-02-22T21:41:51.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fba7415af60 con 0x7fba7415c710 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fba7415b240 con 0x7fba74150220 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.557+0000 7fba7b4bd640 10 monclient(hunting): authenticate will time out at 1387.137604s 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fba7415c710 0x7fba7406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fba74150220 0x7fba740c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba78a31640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fba74150600 0x7fba7415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:38854/0 (socket says 10.20.193.51:38854) 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba78a31640 1 -- 10.20.193.51:0/1917444926 learned_addr learned my addr 10.20.193.51:0/1917444926 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): get_auth_request con 0x7fba7415c710 auth_method 0 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:51.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): get_auth_request con 0x7fba74150220 auth_method 0 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:51.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.558+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 1 -- 10.20.193.51:0/1917444926 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732436181 0 0) 0x7fba7415a920 con 0x7fba74150600 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 10 monclient(hunting): _negotiate 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 10 monclient(hunting): my global_id is 5015 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba63fff640 1 -- 10.20.193.51:0/1917444926 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba5c003620 con 0x7fba74150600 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 10 monclient(hunting): handle_auth_done global_id 5052 payload 274 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 1 -- 10.20.193.51:0/1917444926 >> v2:10.20.193.68:3300/0 conn(0x7fba74150220 msgr2=0x7fba740c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:51.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 1 --2- 10.20.193.51:0/1917444926 >> v2:10.20.193.68:3300/0 conn(0x7fba74150220 0x7fba740c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 1 -- 10.20.193.51:0/1917444926 >> v1:10.20.193.125:6789/0 conn(0x7fba74150600 legacy=0x7fba7415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 10 monclient: _finish_hunting 0 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 1 monclient: found mon.noname-a 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.559+0000 7fba79a33640 1 -- 10.20.193.51:0/1917444926 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7415b520 con 0x7fba7415c710 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.560+0000 7fba79a33640 1 --2- 10.20.193.51:0/1917444926 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba7406fca0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fba74070940 tx=0x7fba7002f250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe3d6532e8bd2b9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 1 -- 10.20.193.51:0/1917444926 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7003c070 con 0x7fba7415c710 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: dump: 2026-02-22T21:41:51.572 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: _finish_auth 0 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:51.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 1 -- 10.20.193.51:0/1917444926 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba70037440 con 0x7fba7415c710 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba7b4bd640 5 monclient: authenticate success, global_id 5052 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba7b4bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba7b4bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 1 -- 10.20.193.51:0/1917444926 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70037810 con 0x7fba7415c710 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba7b4bd640 4 set_mon_vals no callback set 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.562+0000 7fba63fff640 10 monclient: dump: 2026-02-22T21:41:51.574 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 10 monclient: shutdown 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:51.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 -- 10.20.193.51:0/1917444926 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 msgr2=0x7fba7406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/1917444926 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba7406fca0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fba74070940 tx=0x7fba7002f250 comp rx=0 tx=0).stop 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 -- 10.20.193.51:0/1917444926 shutdown_connections 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/1917444926 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba7406fca0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/1917444926 >> v2:10.20.193.68:3300/0 conn(0x7fba74150220 0x7fba740c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 -- 10.20.193.51:0/1917444926 >> 10.20.193.51:0/1917444926 conn(0x7fba740cd7b0 msgr2=0x7fba740cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 -- 10.20.193.51:0/1917444926 shutdown_connections 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 -- 10.20.193.51:0/1917444926 wait complete. 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 10 monclient: build_initial_monmap 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 10 monclient: monmap: 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:51.576 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 librados: starting msgr at 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.563+0000 7fba7b4bd640 1 librados: starting objecter 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 Processor -- start 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 -- start start 2026-02-22T21:41:51.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 librados: setting wanted keys 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 librados: calling monclient init 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient: init 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fba74150220 0x7fba7414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-b con 0x7fba74150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fba74150600 0x7fba7406d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-c con 0x7fba74150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): picked mon.noname-a con 0x7fba7415c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:51.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fba740701e0 con 0x7fba7415c710 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fba7415bae0 con 0x7fba74150220 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fba74070360 con 0x7fba74150600 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba7b4bd640 10 monclient(hunting): authenticate will time out at 1387.144605s 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:49460/0 (socket says 10.20.193.51:49460) 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79232640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fba74150220 0x7fba7414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 1 -- 10.20.193.51:0/3415038031 learned_addr learned my addr 10.20.193.51:0/3415038031 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 10 monclient(hunting): get_auth_request con 0x7fba7415c710 auth_method 0 2026-02-22T21:41:51.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.564+0000 7fba79a33640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): get_auth_request con 0x7fba74150220 auth_method 0 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.565+0000 7fba79232640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:51.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 10 monclient(hunting): handle_auth_done global_id 5055 payload 824 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 -- 10.20.193.51:0/3415038031 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fba74150600 msgr2=0x7fba7406d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fba74150600 0x7fba7406d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 -- 10.20.193.51:0/3415038031 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fba74150220 msgr2=0x7fba7414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fba74150220 0x7fba7414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 10 monclient: _finish_hunting 0 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 monclient: found mon.noname-a 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.566+0000 7fba79a33640 1 -- 10.20.193.51:0/3415038031 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7414d420 con 0x7fba7415c710 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.567+0000 7fba79a33640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba7002f780 tx=0x7fba70004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.567+0000 7fba61ffb640 10 client.?.objecter ms_handle_connect 0x7fba7415c710 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.567+0000 7fba61ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7003c050 con 0x7fba7415c710 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:51.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: dump: 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: _finish_auth 0 2026-02-22T21:41:51.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba70048070 con 0x7fba7415c710 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70041a50 con 0x7fba7415c710 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 5 monclient: authenticate success, global_id 5055 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba637fe640 4 set_mon_vals no callback set 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _renew_subs 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba740c01a0 con 0x7fba7415c710 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:51.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba61ffb640 10 monclient: dump: 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 client.5055.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:51.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _renew_subs 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba7406e7c0 con 0x7fba7415c710 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 1 librados: init done 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.569+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7406fca0 con 0x7fba7415c710 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fba70041bf0 con 0x7fba7415c710 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba61ffb640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 0x7fba4407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba79232640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 0x7fba4407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba79232640 10 monclient: get_auth_request con 0x7fba44078e50 auth_method 0 2026-02-22T21:41:51.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba61ffb640 10 client.5055.objecter ms_handle_connect 0x7fba44078e50 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.581+0000 7fba79232640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 0x7fba4407b310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba64009c40 tx=0x7fba64009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.583+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fba70084370 con 0x7fba7415c710 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.583+0000 7fba61ffb640 10 client.5055.objecter ms_dispatch2 0x7fba74000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.583+0000 7fba61ffb640 3 client.5055.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.583+0000 7fba61ffb640 3 client.5055.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:51.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.583+0000 7fba61ffb640 20 client.5055.objecter dump_active .. 0 homeless 2026-02-22T21:41:51.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.704+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fba7414d420 con 0x7fba7415c710 2026-02-22T21:41:51.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.704+0000 7fba61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:51.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.704+0000 7fba61ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:51.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.804+0000 7fba7b4bd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:51.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.804+0000 7fba7b4bd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:51.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.804+0000 7fba7b4bd640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:51.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.804+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fba7415e0d0 con 0x7fba7415c710 2026-02-22T21:41:51.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.807+0000 7fba61ffb640 1 -- 10.20.193.51:0/3415038031 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fba70088a30 con 0x7fba7415c710 2026-02-22T21:41:51.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.807+0000 7fba61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:51.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.807+0000 7fba61ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:51.808 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:51.809 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 10 librados: watch_flush enter 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 10 librados: watch_flush exit 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 20 client.5055.objecter shutdown clearing up homeless session... 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 10 client.5055.objecter successfully canceled tick 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 msgr2=0x7fba4407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 0x7fba4407b310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba64009c40 tx=0x7fba64009290 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 10 monclient: shutdown 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 msgr2=0x7fba74073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba7002f780 tx=0x7fba70004b40 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 shutdown_connections 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fba44078e50 0x7fba4407b310 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fba7415c710 0x7fba74073420 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fba74150600 0x7fba7406d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 --2- 10.20.193.51:0/3415038031 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fba74150220 0x7fba7414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 >> 10.20.193.51:0/3415038031 conn(0x7fba740cd7b0 msgr2=0x7fba740cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 shutdown_connections 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 -- 10.20.193.51:0/3415038031 wait complete. 2026-02-22T21:41:51.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:51.808+0000 7fba7b4bd640 1 librados: shutdown 2026-02-22T21:41:54.819 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.898+0000 7feeff3b3640 10 monclient: get_monmap_and_config 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.898+0000 7feeff3b3640 10 monclient: build_initial_monmap 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.898+0000 7feeff3b3640 10 monclient: monmap: 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:54.899639+0000 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:54.899639+0000 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:54.900 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 Processor -- start 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 -- start start 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient: init 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient: _add_conns ranks=[3,4,5,1,0,2] 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7feef8150220 0x7feef80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-a con 0x7feef8150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7feef8150600 0x7feef815c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-b con 0x7feef8150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7feef815c540 0x7feef806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-c con 0x7feef815c540 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7feef815c9b0 con 0x7feef8150220 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7feef815a920 con 0x7feef8150600 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7feef815af60 con 0x7feef815c540 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.899+0000 7feeff3b3640 10 monclient(hunting): authenticate will time out at 1390.479951s 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7feef8150220 0x7feef80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7feef8150220 0x7feef80c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:49470/0 (socket says 10.20.193.51:49470) 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 1 -- 10.20.193.51:0/3088458103 learned_addr learned my addr 10.20.193.51:0/3088458103 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): get_auth_request con 0x7feef8150220 auth_method 0 2026-02-22T21:41:54.901 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.902 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefc927640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.68:3300/0 conn(0x7feef8150600 0x7feef815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd929640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.125:3300/0 conn(0x7feef815c540 0x7feef806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefc927640 10 monclient(hunting): get_auth_request con 0x7feef8150600 auth_method 0 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefc927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefc927640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefc927640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd929640 10 monclient(hunting): get_auth_request con 0x7feef815c540 auth_method 0 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd929640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd929640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.900+0000 7feefd929640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 10 monclient(hunting): handle_auth_done global_id 5058 payload 274 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 -- 10.20.193.51:0/3088458103 >> v2:10.20.193.125:3300/0 conn(0x7feef815c540 msgr2=0x7feef806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.125:3300/0 conn(0x7feef815c540 0x7feef806fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 -- 10.20.193.51:0/3088458103 >> v2:10.20.193.68:3300/0 conn(0x7feef8150600 msgr2=0x7feef815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.68:3300/0 conn(0x7feef8150600 0x7feef815c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 10 monclient: _finish_hunting 0 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 monclient: found mon.noname-a 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.901+0000 7feefd128640 1 -- 10.20.193.51:0/3088458103 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef815b240 con 0x7feef8150220 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.902+0000 7feefd128640 1 --2- 10.20.193.51:0/3088458103 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef80c73a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feef8074940 tx=0x7feee802ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5930380b770314 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 1 -- 10.20.193.51:0/3088458103 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee803c070 con 0x7feef8150220 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: dump: 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: _finish_auth 0 2026-02-22T21:41:54.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 1 -- 10.20.193.51:0/3088458103 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee802fbd0 con 0x7feef8150220 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 1 -- 10.20.193.51:0/3088458103 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8037440 con 0x7feef8150220 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 5 monclient: authenticate success, global_id 5058 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 4 set_mon_vals no callback set 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feee7fff640 10 monclient: dump: 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 10 monclient: shutdown 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 -- 10.20.193.51:0/3088458103 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 msgr2=0x7feef80c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 --2- 10.20.193.51:0/3088458103 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef80c73a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feef8074940 tx=0x7feee802ef50 comp rx=0 tx=0).stop 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 -- 10.20.193.51:0/3088458103 shutdown_connections 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.125:3300/0 conn(0x7feef815c540 0x7feef806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 --2- 10.20.193.51:0/3088458103 >> v2:10.20.193.68:3300/0 conn(0x7feef8150600 0x7feef815c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 --2- 10.20.193.51:0/3088458103 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef80c73a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 -- 10.20.193.51:0/3088458103 >> 10.20.193.51:0/3088458103 conn(0x7feef80cd7b0 msgr2=0x7feef80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 -- 10.20.193.51:0/3088458103 shutdown_connections 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.904+0000 7feeff3b3640 1 -- 10.20.193.51:0/3088458103 wait complete. 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 10 monclient: build_initial_monmap 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 10 monclient: monmap: 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:54.907 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 librados: starting msgr at 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 librados: starting objecter 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 Processor -- start 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 -- start start 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 librados: setting wanted keys 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 1 librados: calling monclient init 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.905+0000 7feeff3b3640 10 monclient: init 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:54.908 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-a con 0x7feef8150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feef8150600 0x7feef8156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-b con 0x7feef8150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feef815c540 0x7feef814c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): picked mon.noname-c con 0x7feef815c540 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7feef815bae0 con 0x7feef8150220 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7feef815b800 con 0x7feef8150600 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7feef80701e0 con 0x7feef815c540 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feeff3b3640 10 monclient(hunting): authenticate will time out at 1390.486304s 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:49480/0 (socket says 10.20.193.51:49480) 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 1 -- 10.20.193.51:0/960189112 learned_addr learned my addr 10.20.193.51:0/960189112 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): get_auth_request con 0x7feef8150220 auth_method 0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd929640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feef815c540 0x7feef814c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefc927640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feef8150600 0x7feef8156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd128640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefc927640 10 monclient(hunting): get_auth_request con 0x7feef8150600 auth_method 0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefc927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefc927640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefc927640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd929640 10 monclient(hunting): get_auth_request con 0x7feef815c540 auth_method 0 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd929640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd929640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.906+0000 7feefd929640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefc927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:54.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.907+0000 7feefd929640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 10 monclient(hunting): handle_auth_done global_id 5061 payload 824 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 -- 10.20.193.51:0/960189112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feef815c540 msgr2=0x7feef814c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feef815c540 0x7feef814c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 -- 10.20.193.51:0/960189112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feef8150600 msgr2=0x7feef8156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feef8150600 0x7feef8156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 10 monclient: _finish_hunting 0 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 monclient: found mon.noname-a 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.908+0000 7feefd128640 1 -- 10.20.193.51:0/960189112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef80c1e40 con 0x7feef8150220 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.909+0000 7feefd128640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feee8009a40 tx=0x7feee80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.909+0000 7feee5ffb640 10 client.?.objecter ms_handle_connect 0x7feef8150220 2026-02-22T21:41:54.911 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.909+0000 7feee5ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.910+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee803c040 con 0x7feef8150220 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.910+0000 7feee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: dump: 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: _finish_auth 0 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8042430 con 0x7feef8150220 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8042800 con 0x7feef8150220 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee77fe640 4 set_mon_vals no callback set 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feee5ffb640 10 monclient: dump: 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:54.912 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 5 monclient: authenticate success, global_id 5061 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _renew_subs 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef8157000 con 0x7feef8150220 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 client.5061.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _renew_subs 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:54.913 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef81f18a0 con 0x7feef8150220 2026-02-22T21:41:54.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 1 librados: init done 2026-02-22T21:41:54.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:54.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:54.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:54.914 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.911+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef8070570 con 0x7feef8150220 2026-02-22T21:41:54.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.923+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7feee80429a0 con 0x7feef8150220 2026-02-22T21:41:54.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.924+0000 7feee5ffb640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 0x7feec807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:54.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.924+0000 7feefc927640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 0x7feec807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:54.925 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.924+0000 7feefc927640 10 monclient: get_auth_request con 0x7feec8078e30 auth_method 0 2026-02-22T21:41:54.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.924+0000 7feee5ffb640 10 client.5061.objecter ms_handle_connect 0x7feec8078e30 2026-02-22T21:41:54.926 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:54.924+0000 7feefc927640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 0x7feec807b2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feef80c05a0 tx=0x7feeec009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7feee8083700 con 0x7feef8150220 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 10 client.5061.objecter ms_dispatch2 0x7feef8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 3 client.5061.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 3 client.5061.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 20 client.5061.objecter dump_active .. 0 homeless 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7feee80876c0 con 0x7feef8150220 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:55.029 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.027+0000 7feee5ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:55.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.126+0000 7feeff3b3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:55.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.127+0000 7feeff3b3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:55.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.127+0000 7feeff3b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:41:55.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.127+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7feef81568a0 con 0x7feef8150220 2026-02-22T21:41:55.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.129+0000 7feee5ffb640 1 -- 10.20.193.51:0/960189112 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7feee808b9c0 con 0x7feef8150220 2026-02-22T21:41:55.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.129+0000 7feee5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:55.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.129+0000 7feee5ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:55.131 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:55.131 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 10 librados: watch_flush enter 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 10 librados: watch_flush exit 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 20 client.5061.objecter shutdown clearing up homeless session... 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 10 client.5061.objecter successfully canceled tick 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 msgr2=0x7feec807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 0x7feec807b2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feef80c05a0 tx=0x7feeec009290 comp rx=0 tx=0).stop 2026-02-22T21:41:55.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 10 monclient: shutdown 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 msgr2=0x7feef81f52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feee8009a40 tx=0x7feee80047c0 comp rx=0 tx=0).stop 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 shutdown_connections 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7feec8078e30 0x7feec807b2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7feef815c540 0x7feef814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7feef8150600 0x7feef8156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 --2- 10.20.193.51:0/960189112 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7feef8150220 0x7feef81f52e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 >> 10.20.193.51:0/960189112 conn(0x7feef80cd7b0 msgr2=0x7feef806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 shutdown_connections 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 -- 10.20.193.51:0/960189112 wait complete. 2026-02-22T21:41:55.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:55.131+0000 7feeff3b3640 1 librados: shutdown 2026-02-22T21:41:58.142 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.209+0000 7f1182cc4640 10 monclient: get_monmap_and_config 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.209+0000 7f1182cc4640 10 monclient: build_initial_monmap 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.209+0000 7f1182cc4640 10 monclient: monmap: 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:41:58.210385+0000 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:41:58.210385+0000 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.211 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:41:58.212 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 Processor -- start 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 -- start start 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient: init 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient: _add_conns ranks=[2,5,4,1,3,0] 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f117c150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f117c15c710 0x7f117c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-c con 0x7f117c15c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f117c150600 0x7f117c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.214 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-b con 0x7f117c150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f117c15a920 con 0x7f117c150220 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f117c15af60 con 0x7f117c150600 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f117c15b240 con 0x7f117c15c710 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.210+0000 7f1182cc4640 10 monclient(hunting): authenticate will time out at 1393.790777s 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f117c150600 0x7f117c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1173fff640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f117c15c710 0x7f117c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1180a39640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f117c150220 0x7f117c15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:35032/0 (socket says 10.20.193.51:35032) 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1180a39640 1 -- 10.20.193.51:0/261307001 learned_addr learned my addr 10.20.193.51:0/261307001 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): get_auth_request con 0x7f117c150600 auth_method 0 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1173fff640 10 monclient(hunting): get_auth_request con 0x7f117c15c710 auth_method 0 2026-02-22T21:41:58.215 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1173fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f1173fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.211+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 10 monclient(hunting): handle_auth_done global_id 5068 payload 274 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 1 -- 10.20.193.51:0/261307001 >> v2:10.20.193.125:3300/0 conn(0x7f117c15c710 msgr2=0x7f117c0c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 1 --2- 10.20.193.51:0/261307001 >> v2:10.20.193.125:3300/0 conn(0x7f117c15c710 0x7f117c0c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 1 -- 10.20.193.51:0/261307001 >> v1:10.20.193.125:6789/0 conn(0x7f117c150220 legacy=0x7f117c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 10 monclient: _finish_hunting 0 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 1 monclient: found mon.noname-b 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:41:58.216 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.212+0000 7f118123a640 1 -- 10.20.193.51:0/261307001 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117c15b520 con 0x7f117c150600 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.213+0000 7f118123a640 1 --2- 10.20.193.51:0/261307001 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150600 0x7f117c06fca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1178009d90 tx=0x7f117802f270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5761c6304ab24df server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 1 -- 10.20.193.51:0/261307001 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117803c070 con 0x7f117c150600 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 10 monclient: dump: 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 10 monclient: _finish_auth 0 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 10 monclient: _check_auth_tickets 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f11737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f1182cc4640 5 monclient: authenticate success, global_id 5068 2026-02-22T21:41:58.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.214+0000 7f1182cc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 1 -- 10.20.193.51:0/261307001 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1178037440 con 0x7f117c150600 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 1 -- 10.20.193.51:0/261307001 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178037810 con 0x7f117c150600 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f11737fe640 10 monclient: dump: 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 10 monclient: get_monmap_and_config success 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 4 set_mon_vals no callback set 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 10 monclient: shutdown 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 -- 10.20.193.51:0/261307001 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150600 msgr2=0x7f117c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 --2- 10.20.193.51:0/261307001 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150600 0x7f117c06fca0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1178009d90 tx=0x7f117802f270 comp rx=0 tx=0).stop 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 -- 10.20.193.51:0/261307001 shutdown_connections 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 --2- 10.20.193.51:0/261307001 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150600 0x7f117c06fca0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 --2- 10.20.193.51:0/261307001 >> v2:10.20.193.125:3300/0 conn(0x7f117c15c710 0x7f117c0c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 -- 10.20.193.51:0/261307001 >> 10.20.193.51:0/261307001 conn(0x7f117c0cd7b0 msgr2=0x7f117c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 -- 10.20.193.51:0/261307001 shutdown_connections 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.215+0000 7f1182cc4640 1 -- 10.20.193.51:0/261307001 wait complete. 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient: build_initial_monmap 2026-02-22T21:41:58.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient: monmap: 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 librados: starting msgr at 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 librados: starting objecter 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 Processor -- start 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 -- start start 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 librados: setting wanted keys 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 librados: calling monclient init 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient: init 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient: _reopen_session rank -1 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150220 0x7f117c1f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-b con 0x7f117c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-c con 0x7f117c150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f117c15c710 0x7f117c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): picked mon.noname-a con 0x7f117c15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f117c0701e0 con 0x7f117c15c710 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f117c15bae0 con 0x7f117c150220 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:41:58.221 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f117c070360 con 0x7f117c150600 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): _renew_subs 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.216+0000 7f1182cc4640 10 monclient(hunting): authenticate will time out at 1393.796956s 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150220 0x7f117c1f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f117c15c710 0x7f117c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:55102/0 (socket says 10.20.193.51:55102) 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 1 -- 10.20.193.51:0/806149800 learned_addr learned my addr 10.20.193.51:0/806149800 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): get_auth_request con 0x7f117c150600 auth_method 0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): get_auth_request con 0x7f117c150220 auth_method 0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): get_auth_request con 0x7f117c15c710 auth_method 0 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f1180a39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:41:58.222 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.217+0000 7f118123a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:41:58.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 10 monclient(hunting): handle_auth_done global_id 5030 payload 824 2026-02-22T21:41:58.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 -- 10.20.193.51:0/806149800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150220 msgr2=0x7f117c1f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150220 0x7f117c1f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 -- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f117c15c710 msgr2=0x7f117c06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f117c15c710 0x7f117c06e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 10 monclient: _finish_hunting 0 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 monclient: found mon.noname-c 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.218+0000 7f1173fff640 1 -- 10.20.193.51:0/806149800 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117c1f4530 con 0x7f117c150600 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.219+0000 7f1173fff640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f116400e890 tx=0x7f116400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.220+0000 7f11717fa640 10 client.?.objecter ms_handle_connect 0x7f117c150600 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.220+0000 7f11717fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164017070 con 0x7f117c150600 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: dump: 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: _finish_auth 0 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: _check_auth_tickets 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 5 monclient: authenticate success, global_id 5030 2026-02-22T21:41:58.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _renew_subs 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117c158730 con 0x7f117c150600 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 client.5030.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _renew_subs 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f117c1f7a10 con 0x7f117c150600 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 1 librados: init done 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1164005a90 con 0x7f117c150600 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1172ffd640 4 set_mon_vals no callback set 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164021060 con 0x7f117c150600 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f11717fa640 10 monclient: dump: 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:58.225 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.221+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f117c1f3a10 con 0x7f117c150600 2026-02-22T21:41:58.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.233+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1164024060 con 0x7f117c150600 2026-02-22T21:41:58.234 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.233+0000 7f11717fa640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 0x7f114c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:41:58.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.233+0000 7f1180a39640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 0x7f114c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:41:58.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.233+0000 7f1180a39640 10 monclient: get_auth_request con 0x7f114c078e00 auth_method 0 2026-02-22T21:41:58.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.234+0000 7f11717fa640 10 client.5030.objecter ms_handle_connect 0x7f114c078e00 2026-02-22T21:41:58.235 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.234+0000 7f1180a39640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 0x7f114c07b2c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f116c002770 tx=0x7f116c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:41:58.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.235+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f11640976b0 con 0x7f117c150600 2026-02-22T21:41:58.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.235+0000 7f11717fa640 10 client.5030.objecter ms_dispatch2 0x7f117c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:41:58.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.235+0000 7f11717fa640 3 client.5030.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:41:58.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.235+0000 7f11717fa640 3 client.5030.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:41:58.236 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.235+0000 7f11717fa640 20 client.5030.objecter dump_active .. 0 homeless 2026-02-22T21:41:58.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.339+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f117c1f3a10 con 0x7f117c150600 2026-02-22T21:41:58.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.339+0000 7f11717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:41:58.341 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.339+0000 7f11717fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:41:58.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.439+0000 7f1182cc4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:41:58.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.439+0000 7f1182cc4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:58.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.439+0000 7f1182cc4640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:41:58.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.439+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f117c06fca0 con 0x7f117c150600 2026-02-22T21:41:58.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f11717fa640 1 -- 10.20.193.51:0/806149800 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f1164061ce0 con 0x7f117c150600 2026-02-22T21:41:58.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f11717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:41:58.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f11717fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:41:58.444 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:41:58.444 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:41:58.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f1182cc4640 10 librados: watch_flush enter 2026-02-22T21:41:58.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f1182cc4640 10 librados: watch_flush exit 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f1182cc4640 20 client.5030.objecter shutdown clearing up homeless session... 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.442+0000 7f1182cc4640 10 client.5030.objecter successfully canceled tick 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 msgr2=0x7f114c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 0x7f114c07b2c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f116c002770 tx=0x7f116c009040 comp rx=0 tx=0).stop 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 10 monclient: shutdown 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 msgr2=0x7f117c1493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f116400e890 tx=0x7f116400ed60 comp rx=0 tx=0).stop 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 shutdown_connections 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f114c078e00 0x7f114c07b2c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f117c15c710 0x7f117c06e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f117c150600 0x7f117c1493f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 --2- 10.20.193.51:0/806149800 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f117c150220 0x7f117c1f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 >> 10.20.193.51:0/806149800 conn(0x7f117c0cd7b0 msgr2=0x7f117c15cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 shutdown_connections 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 -- 10.20.193.51:0/806149800 wait complete. 2026-02-22T21:41:58.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:41:58.443+0000 7f1182cc4640 1 librados: shutdown 2026-02-22T21:42:01.453 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:01.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 10 monclient: get_monmap_and_config 2026-02-22T21:42:01.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 10 monclient: build_initial_monmap 2026-02-22T21:42:01.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 10 monclient: monmap: 2026-02-22T21:42:01.526 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:01.522195+0000 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:01.522195+0000 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.527 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:01.528 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 1 Processor -- start 2026-02-22T21:42:01.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 1 -- start start 2026-02-22T21:42:01.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.521+0000 7fb5b5a57640 10 monclient: init 2026-02-22T21:42:01.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:01.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient: _add_conns ranks=[0,2,3,5,4,1] 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5b0150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5b015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb5b00703b0 0x7fb5b00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-a con 0x7fb5b00703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b015a920 con 0x7fb5b0150220 2026-02-22T21:42:01.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b015af60 con 0x7fb5b015c710 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fb5b015b240 con 0x7fb5b00703b0 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5b5a57640 10 monclient(hunting): authenticate will time out at 1397.102491s 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.522+0000 7fb5af7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb5b00703b0 0x7fb5b00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:01.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5ae7fc640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fb5b015c710 0x7fb5b006fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:35038/0 (socket says 10.20.193.51:35038) 2026-02-22T21:42:01.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5ae7fc640 1 -- 10.20.193.51:0/1042860553 learned_addr learned my addr 10.20.193.51:0/1042860553 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:01.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request con 0x7fb5b00703b0 auth_method 0 2026-02-22T21:42:01.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:01.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:01.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:01.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:01.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:01.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.523+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:01.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.524+0000 7fb5adffb640 1 -- 10.20.193.51:0/1042860553 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670275507 0 0) 0x7fb5b015af60 con 0x7fb5b015c710 2026-02-22T21:42:01.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.524+0000 7fb5adffb640 10 monclient(hunting): _negotiate 2026-02-22T21:42:01.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.524+0000 7fb5adffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.524+0000 7fb5adffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.524+0000 7fb5adffb640 10 monclient(hunting): my global_id is 5033 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5adffb640 1 -- 10.20.193.51:0/1042860553 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb59c003620 con 0x7fb5b015c710 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_done global_id 5067 payload 274 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 1 -- 10.20.193.51:0/1042860553 >> v1:10.20.193.125:6789/0 conn(0x7fb5b015c710 legacy=0x7fb5b006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 1 -- 10.20.193.51:0/1042860553 >> v1:10.20.193.51:6789/0 conn(0x7fb5b0150220 legacy=0x7fb5b015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:01.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.525+0000 7fb5af7fe640 1 -- 10.20.193.51:0/1042860553 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b015b520 con 0x7fb5b00703b0 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.527+0000 7fb5af7fe640 1 --2- 10.20.193.51:0/1042860553 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b00703b0 0x7fb5b00c73a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009dc0 tx=0x7fb5a002f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6b24eaf4dea569 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 1 -- 10.20.193.51:0/1042860553 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a003c070 con 0x7fb5b00703b0 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:01.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: dump: 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:01.549 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 1 -- 10.20.193.51:0/1042860553 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a0037440 con 0x7fb5b00703b0 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 5 monclient: authenticate success, global_id 5067 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 1 -- 10.20.193.51:0/1042860553 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0037810 con 0x7fb5b00703b0 2026-02-22T21:42:01.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 4 set_mon_vals no callback set 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5adffb640 10 monclient: dump: 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.551 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 10 monclient: shutdown 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 -- 10.20.193.51:0/1042860553 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b00703b0 msgr2=0x7fb5b00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/1042860553 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b00703b0 0x7fb5b00c73a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009dc0 tx=0x7fb5a002f2f0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 -- 10.20.193.51:0/1042860553 shutdown_connections 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/1042860553 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b00703b0 0x7fb5b00c73a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 -- 10.20.193.51:0/1042860553 >> 10.20.193.51:0/1042860553 conn(0x7fb5b00cd7b0 msgr2=0x7fb5b00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 -- 10.20.193.51:0/1042860553 shutdown_connections 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.529+0000 7fb5b5a57640 1 -- 10.20.193.51:0/1042860553 wait complete. 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient: build_initial_monmap 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient: monmap: 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:01.553 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 librados: starting msgr at 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 librados: starting objecter 2026-02-22T21:42:01.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 Processor -- start 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 -- start start 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 librados: setting wanted keys 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 librados: calling monclient init 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient: init 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5b00703b0 0x7fb5b006e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b00703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5b0150220 0x7fb5b01eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:01.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): picked mon.noname-a con 0x7fb5b015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b0150600 con 0x7fb5b015c710 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b015bae0 con 0x7fb5b00703b0 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b0150780 con 0x7fb5b0150220 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5b5a57640 10 monclient(hunting): authenticate will time out at 1397.110724s 2026-02-22T21:42:01.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.530+0000 7fb5af7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35804/0 (socket says 10.20.193.51:35804) 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 1 -- 10.20.193.51:0/3956517391 learned_addr learned my addr 10.20.193.51:0/3956517391 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request con 0x7fb5b015c710 auth_method 0 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5b0150220 0x7fb5b01eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:01.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5b00703b0 0x7fb5b006e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb5b0150220 auth_method 0 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request con 0x7fb5b00703b0 auth_method 0 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:01.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.531+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_done global_id 5073 payload 824 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 -- 10.20.193.51:0/3956517391 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5b0150220 msgr2=0x7fb5b01eb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5b0150220 0x7fb5b01eb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 -- 10.20.193.51:0/3956517391 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5b00703b0 msgr2=0x7fb5b006e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5b00703b0 0x7fb5b006e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:01.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.532+0000 7fb5af7fe640 1 -- 10.20.193.51:0/3956517391 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b006d520 con 0x7fb5b015c710 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.533+0000 7fb5af7fe640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00172c0 tx=0x7fb5a0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.533+0000 7fb587fff640 10 client.?.objecter ms_handle_connect 0x7fb5b015c710 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.533+0000 7fb587fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a003c050 con 0x7fb5b015c710 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: dump: 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.560 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a0048070 con 0x7fb5b015c710 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:01.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0041a50 con 0x7fb5b015c710 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 5 monclient: authenticate success, global_id 5073 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _renew_subs 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b006d800 con 0x7fb5b015c710 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5ad7fa640 4 set_mon_vals no callback set 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb587fff640 10 monclient: dump: 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.562 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 client.5073.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _renew_subs 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b006ddc0 con 0x7fb5b015c710 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 1 librados: init done 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:01.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:01.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.535+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b0074da0 con 0x7fb5b015c710 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb5a0041bf0 con 0x7fb5b015c710 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb587fff640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 0x7fb58007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb5aeffd640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 0x7fb58007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb5aeffd640 10 monclient: get_auth_request con 0x7fb580078e50 auth_method 0 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb587fff640 10 client.5073.objecter ms_handle_connect 0x7fb580078e50 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.547+0000 7fb5aeffd640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 0x7fb58007b310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4002770 tx=0x7fb5a4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.549+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb5a0084390 con 0x7fb5b015c710 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.549+0000 7fb587fff640 10 client.5073.objecter ms_dispatch2 0x7fb5b0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.549+0000 7fb587fff640 3 client.5073.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:01.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.549+0000 7fb587fff640 3 client.5073.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:01.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.549+0000 7fb587fff640 20 client.5073.objecter dump_active .. 0 homeless 2026-02-22T21:42:01.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.652+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fb5a0088350 con 0x7fb5b015c710 2026-02-22T21:42:01.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.652+0000 7fb587fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:01.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.652+0000 7fb587fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:01.682 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:32.836 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.709 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:32.861 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.710 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:32.863 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.710 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:32.826 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.711 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:32.863 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.712 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:32.827 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.720 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:32.872 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.721 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:32.875 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.721 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:32.838 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:01.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.756+0000 7fb5b5a57640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:01.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.756+0000 7fb5b5a57640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:01.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.756+0000 7fb5b5a57640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:01.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.756+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb5b00c5420 con 0x7fb5b015c710 2026-02-22T21:42:01.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.760+0000 7fb587fff640 1 -- 10.20.193.51:0/3956517391 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fb5a008c650 con 0x7fb5b015c710 2026-02-22T21:42:01.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.760+0000 7fb587fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:01.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.760+0000 7fb587fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:01.762 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:01.762 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 10 librados: watch_flush enter 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 10 librados: watch_flush exit 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 20 client.5073.objecter shutdown clearing up homeless session... 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 10 client.5073.objecter successfully canceled tick 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 msgr2=0x7fb58007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 0x7fb58007b310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4002770 tx=0x7fb5a4007ef0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 10 monclient: shutdown 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 msgr2=0x7fb5b01ebe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.761+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00172c0 tx=0x7fb5a0004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 shutdown_connections 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb580078e50 0x7fb58007b310 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb5b015c710 0x7fb5b01ebe10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb5b0150220 0x7fb5b01eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 --2- 10.20.193.51:0/3956517391 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb5b00703b0 0x7fb5b006e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 >> 10.20.193.51:0/3956517391 conn(0x7fb5b00cd7b0 msgr2=0x7fb5b00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:01.765 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 shutdown_connections 2026-02-22T21:42:01.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 -- 10.20.193.51:0/3956517391 wait complete. 2026-02-22T21:42:01.766 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:01.762+0000 7fb5b5a57640 1 librados: shutdown 2026-02-22T21:42:04.772 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: get_monmap_and_config 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: build_initial_monmap 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: monmap: 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:04.842750+0000 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:04.842750+0000 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:04.844 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 1 Processor -- start 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 1 -- start start 2026-02-22T21:42:04.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: init 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 10 monclient: _add_conns ranks=[3,5,0,2,4,1] 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.842+0000 7ff482b97640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff47c150220 0x7ff47c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-a con 0x7ff47c150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff47c150600 0x7ff47c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-c con 0x7ff47c150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff47c15c540 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:04.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff47c15c9b0 con 0x7ff47c15c540 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff47c15a920 con 0x7ff47c150220 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ff47c15af60 con 0x7ff47c150600 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:04.848 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.843+0000 7ff482b97640 10 monclient(hunting): authenticate will time out at 1400.423041s 2026-02-22T21:42:04.861 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.860+0000 7ff48090c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff47c150220 0x7ff47c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.860+0000 7ff48110d640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7ff47c15c540 0x7ff47c06fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:56184/0 (socket says 10.20.193.51:56184) 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.860+0000 7ff48110d640 1 -- 10.20.193.51:0/2667108126 learned_addr learned my addr 10.20.193.51:0/2667108126 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): get_auth_request con 0x7ff47c150220 auth_method 0 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:04.862 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:04.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:04.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:04.863 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.861+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 10 monclient(hunting): handle_auth_done global_id 5076 payload 274 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 1 -- 10.20.193.51:0/2667108126 >> v2:10.20.193.125:3300/0 conn(0x7ff47c150600 msgr2=0x7ff47c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 1 --2- 10.20.193.51:0/2667108126 >> v2:10.20.193.125:3300/0 conn(0x7ff47c150600 0x7ff47c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 1 -- 10.20.193.51:0/2667108126 >> v1:10.20.193.51:6789/0 conn(0x7ff47c15c540 legacy=0x7ff47c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 1 monclient: found mon.noname-a 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:04.864 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.862+0000 7ff48090c640 1 -- 10.20.193.51:0/2667108126 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47c15b240 con 0x7ff47c150220 2026-02-22T21:42:04.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.865+0000 7ff48090c640 1 --2- 10.20.193.51:0/2667108126 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c0c73a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff47c074940 tx=0x7ff46c02eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f253c8b36befa917 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 1 -- 10.20.193.51:0/2667108126 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c03c070 con 0x7ff47c150220 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: dump: 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:04.870 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff482b97640 5 monclient: authenticate success, global_id 5076 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 1 -- 10.20.193.51:0/2667108126 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46c02fb80 con 0x7ff47c150220 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 1 -- 10.20.193.51:0/2667108126 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c037440 con 0x7ff47c150220 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff4737fe640 10 monclient: dump: 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.871 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff482b97640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.867+0000 7ff482b97640 4 set_mon_vals no callback set 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 10 monclient: shutdown 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- 10.20.193.51:0/2667108126 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 msgr2=0x7ff47c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 --2- 10.20.193.51:0/2667108126 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c0c73a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff47c074940 tx=0x7ff46c02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- 10.20.193.51:0/2667108126 shutdown_connections 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 --2- 10.20.193.51:0/2667108126 >> v2:10.20.193.125:3300/0 conn(0x7ff47c150600 0x7ff47c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 --2- 10.20.193.51:0/2667108126 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c0c73a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- 10.20.193.51:0/2667108126 >> 10.20.193.51:0/2667108126 conn(0x7ff47c0cd7b0 msgr2=0x7ff47c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- 10.20.193.51:0/2667108126 shutdown_connections 2026-02-22T21:42:04.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- 10.20.193.51:0/2667108126 wait complete. 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 10 monclient: build_initial_monmap 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 10 monclient: monmap: 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 librados: starting msgr at 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 librados: starting objecter 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 Processor -- start 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 -- start start 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 librados: setting wanted keys 2026-02-22T21:42:04.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 1 librados: calling monclient init 2026-02-22T21:42:04.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.868+0000 7ff482b97640 10 monclient: init 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-a con 0x7ff47c150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff47c150600 0x7ff47c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-b con 0x7ff47c150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff47c15c540 0x7ff47c1e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): picked mon.noname-c con 0x7ff47c15c540 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c15bae0 con 0x7ff47c150220 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c15b800 con 0x7ff47c150600 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c0703b0 con 0x7ff47c15c540 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff482b97640 10 monclient(hunting): authenticate will time out at 1400.449488s 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:04.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35828/0 (socket says 10.20.193.51:35828) 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 1 -- 10.20.193.51:0/2832891834 learned_addr learned my addr 10.20.193.51:0/2832891834 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 10 monclient(hunting): get_auth_request con 0x7ff47c150220 auth_method 0 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48090c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff48110d640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff47c15c540 0x7ff47c1e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.869+0000 7ff473fff640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff47c150600 0x7ff47c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48090c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): get_auth_request con 0x7ff47c15c540 auth_method 0 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): get_auth_request con 0x7ff47c150600 auth_method 0 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:04.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff48110d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.870+0000 7ff473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 10 monclient(hunting): handle_auth_done global_id 5082 payload 824 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 -- 10.20.193.51:0/2832891834 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff47c15c540 msgr2=0x7ff47c1e9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff47c15c540 0x7ff47c1e9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 -- 10.20.193.51:0/2832891834 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff47c150600 msgr2=0x7ff47c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff47c150600 0x7ff47c14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 monclient: found mon.noname-a 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.871+0000 7ff48090c640 1 -- 10.20.193.51:0/2832891834 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47c1e7d60 con 0x7ff47c150220 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.872+0000 7ff48090c640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009130 tx=0x7ff46c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.872+0000 7ff4717fa640 10 client.?.objecter ms_handle_connect 0x7ff47c150220 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.872+0000 7ff4717fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c03c040 con 0x7ff47c150220 2026-02-22T21:42:04.879 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 10 monclient: dump: 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.873+0000 7ff4717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46c03f680 con 0x7ff47c150220 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 5 monclient: authenticate success, global_id 5082 2026-02-22T21:42:04.880 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff472ffd640 4 set_mon_vals no callback set 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c03fa50 con 0x7ff47c150220 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _renew_subs 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47c1f3ef0 con 0x7ff47c150220 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff4717fa640 10 monclient: dump: 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 client.5082.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _renew_subs 2026-02-22T21:42:04.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47c1f44b0 con 0x7ff47c150220 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 1 librados: init done 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:04.882 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.874+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff47c15c000 con 0x7ff47c150220 2026-02-22T21:42:04.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.885+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff46c03fbf0 con 0x7ff47c150220 2026-02-22T21:42:04.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.886+0000 7ff4717fa640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 0x7ff44c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:04.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.886+0000 7ff473fff640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 0x7ff44c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:04.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.886+0000 7ff473fff640 10 monclient: get_auth_request con 0x7ff44c078e50 auth_method 0 2026-02-22T21:42:04.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.886+0000 7ff473fff640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 0x7ff44c07b310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff47c154040 tx=0x7ff464009210 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:04.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.886+0000 7ff4717fa640 10 client.5082.objecter ms_handle_connect 0x7ff44c078e50 2026-02-22T21:42:04.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.887+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff46c0839d0 con 0x7ff47c150220 2026-02-22T21:42:04.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.887+0000 7ff4717fa640 10 client.5082.objecter ms_dispatch2 0x7ff47c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:04.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.887+0000 7ff4717fa640 3 client.5082.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:04.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.887+0000 7ff4717fa640 3 client.5082.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:04.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.887+0000 7ff4717fa640 20 client.5082.objecter dump_active .. 0 homeless 2026-02-22T21:42:04.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.988+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff46c087990 con 0x7ff47c150220 2026-02-22T21:42:04.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.988+0000 7ff4717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:04.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:04.988+0000 7ff4717fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:05.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.086+0000 7ff482b97640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:05.088 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.086+0000 7ff482b97640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:05.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.086+0000 7ff482b97640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:05.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.086+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff47c074da0 con 0x7ff47c150220 2026-02-22T21:42:05.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.089+0000 7ff4717fa640 1 -- 10.20.193.51:0/2832891834 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff46c08ce20 con 0x7ff47c150220 2026-02-22T21:42:05.090 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.089+0000 7ff4717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:05.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.089+0000 7ff4717fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:05.091 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:05.091 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 10 librados: watch_flush enter 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 10 librados: watch_flush exit 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 20 client.5082.objecter shutdown clearing up homeless session... 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 10 client.5082.objecter successfully canceled tick 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 msgr2=0x7ff44c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 0x7ff44c07b310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff47c154040 tx=0x7ff464009210 comp rx=0 tx=0).stop 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 10 monclient: shutdown 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:05.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 msgr2=0x7ff47c159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009130 tx=0x7ff46c0047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 shutdown_connections 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff44c078e50 0x7ff44c07b310 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff47c15c540 0x7ff47c1e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff47c150600 0x7ff47c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 --2- 10.20.193.51:0/2832891834 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff47c150220 0x7ff47c159850 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 >> 10.20.193.51:0/2832891834 conn(0x7ff47c0cd7b0 msgr2=0x7ff47c0cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 shutdown_connections 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 -- 10.20.193.51:0/2832891834 wait complete. 2026-02-22T21:42:05.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:05.090+0000 7ff482b97640 1 librados: shutdown 2026-02-22T21:42:08.101 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:08.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:08.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 10 monclient: build_initial_monmap 2026-02-22T21:42:08.168 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 10 monclient: monmap: 2026-02-22T21:42:08.168 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:08.166978+0000 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:08.166978+0000 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:08.169 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:08.170 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 1 Processor -- start 2026-02-22T21:42:08.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 1 -- start start 2026-02-22T21:42:08.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.166+0000 7f3f91f2c640 10 monclient: init 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient: _add_conns ranks=[4,5,2,1,3,0] 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3f8c150220 0x7f3f8c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8c150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f3f8c150600 0x7f3f8c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8c150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f8c15c540 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8c15c9b0 con 0x7f3f8c15c540 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f3f8c15a920 con 0x7f3f8c150220 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f3f8c15af60 con 0x7f3f8c150600 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.167+0000 7f3f91f2c640 10 monclient(hunting): authenticate will time out at 1403.747583s 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3f8c150220 0x7f3f8c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3f8c150220 0x7f3f8c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47666/0 (socket says 10.20.193.51:47666) 2026-02-22T21:42:08.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 1 -- 10.20.193.51:0/1490341419 learned_addr learned my addr 10.20.193.51:0/1490341419 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f8c150220 auth_method 0 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 1 --2- 10.20.193.51:0/1490341419 >> v2:10.20.193.125:3300/0 conn(0x7f3f8c150600 0x7f3f8c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): get_auth_request con 0x7f3f8c150600 auth_method 0 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:08.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.168+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_done global_id 5080 payload 274 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 1 -- 10.20.193.51:0/1490341419 >> v2:10.20.193.125:3300/0 conn(0x7f3f8c150600 msgr2=0x7f3f8c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 1 --2- 10.20.193.51:0/1490341419 >> v2:10.20.193.125:3300/0 conn(0x7f3f8c150600 0x7f3f8c15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 1 -- 10.20.193.51:0/1490341419 >> v1:10.20.193.125:6789/0 conn(0x7f3f8c15c540 legacy=0x7f3f8c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 1 monclient: found mon.noname-b 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.169+0000 7f3f8b7fe640 1 -- 10.20.193.51:0/1490341419 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c15b240 con 0x7f3f8c150220 2026-02-22T21:42:08.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.170+0000 7f3f8b7fe640 1 --2- 10.20.193.51:0/1490341419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c150220 0x7f3f8c0c73a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c074940 tx=0x7f3f7802ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a006df39c639a475 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:08.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 1 -- 10.20.193.51:0/1490341419 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7803c070 con 0x7f3f8c150220 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: dump: 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.184 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:08.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 5 monclient: authenticate success, global_id 5080 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 1 -- 10.20.193.51:0/1490341419 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f7802fc70 con 0x7f3f8c150220 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 1 -- 10.20.193.51:0/1490341419 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78037660 con 0x7f3f8c150220 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 4 set_mon_vals no callback set 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f8a7fc640 10 monclient: dump: 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:08.186 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 10 monclient: shutdown 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1490341419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c150220 msgr2=0x7f3f8c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1490341419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c150220 0x7f3f8c0c73a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c074940 tx=0x7f3f7802ef60 comp rx=0 tx=0).stop 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1490341419 shutdown_connections 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1490341419 >> v2:10.20.193.125:3300/0 conn(0x7f3f8c150600 0x7f3f8c15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1490341419 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c150220 0x7f3f8c0c73a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1490341419 >> 10.20.193.51:0/1490341419 conn(0x7f3f8c0cd7b0 msgr2=0x7f3f8c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1490341419 shutdown_connections 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.172+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1490341419 wait complete. 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient: build_initial_monmap 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient: monmap: 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:08.188 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 librados: starting msgr at 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 librados: starting objecter 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 Processor -- start 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 -- start start 2026-02-22T21:42:08.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 librados: setting wanted keys 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 librados: calling monclient init 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient: init 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3f8c150220 0x7f3f8c14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8c150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3f8c150600 0x7f3f8c0c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-a con 0x7f3f8c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8c15c540 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:08.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c15bae0 con 0x7f3f8c150600 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c15b800 con 0x7f3f8c15c540 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c0703b0 con 0x7f3f8c150220 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.173+0000 7f3f91f2c640 10 monclient(hunting): authenticate will time out at 1403.753906s 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3f8c150600 0x7f3f8c0c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3f8c150220 0x7f3f8c14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.191 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47674/0 (socket says 10.20.193.51:47674) 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 1 -- 10.20.193.51:0/1658801146 learned_addr learned my addr 10.20.193.51:0/1658801146 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): get_auth_request con 0x7f3f8c15c540 auth_method 0 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f8c150220 auth_method 0 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): get_auth_request con 0x7f3f8c150600 auth_method 0 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:08.192 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.174+0000 7f3f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 10 monclient(hunting): handle_auth_done global_id 5083 payload 824 2026-02-22T21:42:08.193 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 -- 10.20.193.51:0/1658801146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3f8c150220 msgr2=0x7f3f8c14c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3f8c150220 0x7f3f8c14c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 -- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3f8c150600 msgr2=0x7f3f8c0c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3f8c150600 0x7f3f8c0c3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 monclient: found mon.noname-b 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.175+0000 7f3f8bfff640 1 -- 10.20.193.51:0/1658801146 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c1f29c0 con 0x7f3f8c15c540 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.176+0000 7f3f8bfff640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000cba0 tx=0x7f3f8000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.176+0000 7f3f67fff640 10 client.?.objecter ms_handle_connect 0x7f3f8c15c540 2026-02-22T21:42:08.194 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.176+0000 7f3f67fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8000ad50 con 0x7f3f8c15c540 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 10 monclient: dump: 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:08.195 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.178+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f80003a70 con 0x7f3f8c15c540 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f67fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f800050d0 con 0x7f3f8c15c540 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f67fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f89ffb640 4 set_mon_vals no callback set 2026-02-22T21:42:08.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f67fff640 10 monclient: dump: 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:08.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 5 monclient: authenticate success, global_id 5083 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _renew_subs 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8c1f5410 con 0x7f3f8c15c540 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 client.5083.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _renew_subs 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8c1f59d0 con 0x7f3f8c15c540 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 1 librados: init done 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:08.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:08.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.179+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8c1f6090 con 0x7f3f8c15c540 2026-02-22T21:42:08.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.190+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f3f80005270 con 0x7f3f8c15c540 2026-02-22T21:42:08.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.191+0000 7f3f67fff640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 0x7f3f5c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:08.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.191+0000 7f3f8b7fe640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 0x7f3f5c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.191+0000 7f3f8b7fe640 10 monclient: get_auth_request con 0x7f3f5c078e00 auth_method 0 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.191+0000 7f3f67fff640 10 client.5083.objecter ms_handle_connect 0x7f3f5c078e00 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.191+0000 7f3f8b7fe640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 0x7f3f5c07b2c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0c65c0 tx=0x7f3f78039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.192+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f3f8005e6c0 con 0x7f3f8c15c540 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.192+0000 7f3f67fff640 10 client.5083.objecter ms_dispatch2 0x7f3f8c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.192+0000 7f3f67fff640 3 client.5083.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.192+0000 7f3f67fff640 3 client.5083.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:08.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.192+0000 7f3f67fff640 20 client.5083.objecter dump_active .. 0 homeless 2026-02-22T21:42:08.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.290+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f3f8c1f6090 con 0x7f3f8c15c540 2026-02-22T21:42:08.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.290+0000 7f3f67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:08.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.290+0000 7f3f67fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:08.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.389+0000 7f3f91f2c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:08.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.389+0000 7f3f91f2c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:08.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.389+0000 7f3f91f2c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:08.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.389+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3f8c15c000 con 0x7f3f8c15c540 2026-02-22T21:42:08.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f67fff640 1 -- 10.20.193.51:0/1658801146 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f3f80062680 con 0x7f3f8c15c540 2026-02-22T21:42:08.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:08.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f67fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:08.394 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:08.394 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 10 librados: watch_flush enter 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 10 librados: watch_flush exit 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 20 client.5083.objecter shutdown clearing up homeless session... 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 10 client.5083.objecter successfully canceled tick 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 msgr2=0x7f3f5c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 0x7f3f5c07b2c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c0c65c0 tx=0x7f3f78039040 comp rx=0 tx=0).stop 2026-02-22T21:42:08.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 10 monclient: shutdown 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 msgr2=0x7f3f8c0c38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000cba0 tx=0x7f3f8000a430 comp rx=0 tx=0).stop 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 shutdown_connections 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3f5c078e00 0x7f3f5c07b2c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3f8c15c540 0x7f3f8c0c38d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3f8c150600 0x7f3f8c0c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 --2- 10.20.193.51:0/1658801146 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3f8c150220 0x7f3f8c14c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 >> 10.20.193.51:0/1658801146 conn(0x7f3f8c0cd7b0 msgr2=0x7f3f8c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 shutdown_connections 2026-02-22T21:42:08.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 -- 10.20.193.51:0/1658801146 wait complete. 2026-02-22T21:42:08.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:08.393+0000 7f3f91f2c640 1 librados: shutdown 2026-02-22T21:42:11.404 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:11.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:11.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 10 monclient: build_initial_monmap 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 10 monclient: monmap: 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:11.469787+0000 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:11.469787+0000 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:11.471 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:11.472 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:11.472 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:11.472 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:11.472 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 1 Processor -- start 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 1 -- start start 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.468+0000 7f184532c640 10 monclient: init 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient: _add_conns ranks=[4,3,5,0,1,2] 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f18401526a0 0x7f18400733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): picked mon.noname-b con 0x7f18401526a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1840152a80 0x7f18400cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): picked mon.noname-a con 0x7f1840152a80 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:11.476 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f18400cdb20 0x7f184015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): picked mon.noname-c con 0x7f18400cdb20 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f18400cca00 con 0x7f1840152a80 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f1840074c50 con 0x7f18401526a0 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f1840073d60 con 0x7f18400cdb20 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f184532c640 10 monclient(hunting): authenticate will time out at 1407.050379s 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1840152a80 0x7f18400cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183effd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f18401526a0 0x7f18400733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.477 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f1840152a80 0x7f18400cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42452/0 (socket says 10.20.193.51:42452) 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 1 -- 10.20.193.51:0/3810419460 learned_addr learned my addr 10.20.193.51:0/3810419460 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 10 monclient(hunting): get_auth_request con 0x7f1840152a80 auth_method 0 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.469+0000 7f183f7fe640 1 --2- 10.20.193.51:0/3810419460 >> v2:10.20.193.125:3300/0 conn(0x7f18400cdb20 0x7f184015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): get_auth_request con 0x7f18401526a0 auth_method 0 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.478 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): get_auth_request con 0x7f18400cdb20 auth_method 0 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.470+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 10 monclient(hunting): handle_auth_done global_id 5088 payload 274 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 -- 10.20.193.51:0/3810419460 >> v2:10.20.193.125:3300/0 conn(0x7f18400cdb20 msgr2=0x7f184015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 --2- 10.20.193.51:0/3810419460 >> v2:10.20.193.125:3300/0 conn(0x7f18400cdb20 0x7f184015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 -- 10.20.193.51:0/3810419460 >> v2:10.20.193.68:3300/0 conn(0x7f18401526a0 msgr2=0x7f18400733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 --2- 10.20.193.51:0/3810419460 >> v2:10.20.193.68:3300/0 conn(0x7f18401526a0 0x7f18400733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 monclient: found mon.noname-a 2026-02-22T21:42:11.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:11.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.471+0000 7f183e7fc640 1 -- 10.20.193.51:0/3810419460 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1840074040 con 0x7f1840152a80 2026-02-22T21:42:11.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.472+0000 7f183e7fc640 1 --2- 10.20.193.51:0/3810419460 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18400cd5e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f184006cf40 tx=0x7f182802ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1548f41bfc013b94 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:11.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 1 -- 10.20.193.51:0/3810419460 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182803c070 con 0x7f1840152a80 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: dump: 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:11.491 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 1 -- 10.20.193.51:0/3810419460 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182802fbd0 con 0x7f1840152a80 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f184532c640 5 monclient: authenticate success, global_id 5088 2026-02-22T21:42:11.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 1 -- 10.20.193.51:0/3810419460 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1828037440 con 0x7f1840152a80 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f184532c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f184532c640 4 set_mon_vals no callback set 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.473+0000 7f183dffb640 10 monclient: dump: 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.493 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 10 monclient: shutdown 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 -- 10.20.193.51:0/3810419460 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 msgr2=0x7f18400cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 --2- 10.20.193.51:0/3810419460 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18400cd5e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f184006cf40 tx=0x7f182802ef50 comp rx=0 tx=0).stop 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 -- 10.20.193.51:0/3810419460 shutdown_connections 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 --2- 10.20.193.51:0/3810419460 >> v2:10.20.193.125:3300/0 conn(0x7f18400cdb20 0x7f184015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 --2- 10.20.193.51:0/3810419460 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18400cd5e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 --2- 10.20.193.51:0/3810419460 >> v2:10.20.193.68:3300/0 conn(0x7f18401526a0 0x7f18400733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 -- 10.20.193.51:0/3810419460 >> 10.20.193.51:0/3810419460 conn(0x7f18400ce350 msgr2=0x7f18400ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 -- 10.20.193.51:0/3810419460 shutdown_connections 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 1 -- 10.20.193.51:0/3810419460 wait complete. 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 10 monclient: build_initial_monmap 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.474+0000 7f184532c640 10 monclient: monmap: 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:11.495 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 librados: starting msgr at 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 librados: starting objecter 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 Processor -- start 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 -- start start 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 librados: setting wanted keys 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 librados: calling monclient init 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient: init 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:11.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f18400cdb20 0x7f18401593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): picked mon.noname-b con 0x7f18400cdb20 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f18401526a0 0x7f184015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): picked mon.noname-c con 0x7f18401526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): picked mon.noname-a con 0x7f1840152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f18400748e0 con 0x7f1840152a80 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1840074600 con 0x7f18400cdb20 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1840073910 con 0x7f18401526a0 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:11.497 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f184532c640 10 monclient(hunting): authenticate will time out at 1407.056516s 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42468/0 (socket says 10.20.193.51:42468) 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 1 -- 10.20.193.51:0/987554679 learned_addr learned my addr 10.20.193.51:0/987554679 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 10 monclient(hunting): get_auth_request con 0x7f1840152a80 auth_method 0 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183effd640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f18400cdb20 0x7f18401593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.475+0000 7f183e7fc640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f18401526a0 0x7f184015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): get_auth_request con 0x7f18400cdb20 auth_method 0 2026-02-22T21:42:11.498 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): get_auth_request con 0x7f18401526a0 auth_method 0 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.499 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.476+0000 7f183e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 10 monclient(hunting): handle_auth_done global_id 5091 payload 824 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 -- 10.20.193.51:0/987554679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f18401526a0 msgr2=0x7f184015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f18401526a0 0x7f184015c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 -- 10.20.193.51:0/987554679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f18400cdb20 msgr2=0x7f18401593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f18400cdb20 0x7f18401593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.500 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.477+0000 7f183f7fe640 1 -- 10.20.193.51:0/987554679 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184015b1c0 con 0x7f1840152a80 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.478+0000 7f183f7fe640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f183000eed0 tx=0x7f183000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.478+0000 7f181f7fe640 10 client.?.objecter ms_handle_connect 0x7f1840152a80 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.478+0000 7f181f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1830018020 con 0x7f1840152a80 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: dump: 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.501 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f183000bd00 con 0x7f1840152a80 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 5 monclient: authenticate success, global_id 5091 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 10 monclient: _renew_subs 2026-02-22T21:42:11.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18400c9680 con 0x7f1840152a80 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1830014640 con 0x7f1840152a80 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f183d7fa640 4 set_mon_vals no callback set 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f181f7fe640 10 monclient: dump: 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:11.503 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 10 client.5091.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 10 monclient: _renew_subs 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18400c9c40 con 0x7f1840152a80 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.479+0000 7f184532c640 1 librados: init done 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.480+0000 7f184532c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.480+0000 7f184532c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.480+0000 7f184532c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.504 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.480+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18400733d0 con 0x7f1840152a80 2026-02-22T21:42:11.505 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.491+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f18300147e0 con 0x7f1840152a80 2026-02-22T21:42:11.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.492+0000 7f181f7fe640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 0x7f180c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:11.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.492+0000 7f183effd640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 0x7f180c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:11.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.492+0000 7f183effd640 10 monclient: get_auth_request con 0x7f180c078e50 auth_method 0 2026-02-22T21:42:11.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.492+0000 7f183effd640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 0x7f180c07b310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1834009ba0 tx=0x7f1834009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:11.506 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.492+0000 7f181f7fe640 10 client.5091.objecter ms_handle_connect 0x7f180c078e50 2026-02-22T21:42:11.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.567+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f183005e9e0 con 0x7f1840152a80 2026-02-22T21:42:11.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.567+0000 7f181f7fe640 10 client.5091.objecter ms_dispatch2 0x7f1840000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:11.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.567+0000 7f181f7fe640 3 client.5091.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:11.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.567+0000 7f181f7fe640 3 client.5091.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:11.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.567+0000 7f181f7fe640 20 client.5091.objecter dump_active .. 0 homeless 2026-02-22T21:42:11.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.599+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f1830022030 con 0x7f1840152a80 2026-02-22T21:42:11.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.599+0000 7f181f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:11.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.599+0000 7f181f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:11.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.699+0000 7f184532c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:11.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.699+0000 7f184532c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:11.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.699+0000 7f184532c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:11.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.699+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f184015cdf0 con 0x7f1840152a80 2026-02-22T21:42:11.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.701+0000 7f181f7fe640 1 -- 10.20.193.51:0/987554679 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f18300629a0 con 0x7f1840152a80 2026-02-22T21:42:11.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.701+0000 7f181f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:11.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.701+0000 7f181f7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:11.704 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:11.704 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 10 librados: watch_flush enter 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 10 librados: watch_flush exit 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 20 client.5091.objecter shutdown clearing up homeless session... 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 10 client.5091.objecter successfully canceled tick 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 msgr2=0x7f180c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:11.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 0x7f180c07b310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1834009ba0 tx=0x7f1834009290 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 10 monclient: shutdown 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 msgr2=0x7f18401f7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f183000eed0 tx=0x7f183000b480 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 shutdown_connections 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f180c078e50 0x7f180c07b310 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1840152a80 0x7f18401f7260 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f18401526a0 0x7f184015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 --2- 10.20.193.51:0/987554679 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f18400cdb20 0x7f18401593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 >> 10.20.193.51:0/987554679 conn(0x7f18400ce350 msgr2=0x7f18400ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:11.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 shutdown_connections 2026-02-22T21:42:11.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 -- 10.20.193.51:0/987554679 wait complete. 2026-02-22T21:42:11.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:11.702+0000 7f184532c640 1 librados: shutdown 2026-02-22T21:42:14.714 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.781+0000 7fc8a6fa0640 10 monclient: get_monmap_and_config 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.781+0000 7fc8a6fa0640 10 monclient: build_initial_monmap 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.781+0000 7fc8a6fa0640 10 monclient: monmap: 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:14.784 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:14.783276+0000 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:14.783276+0000 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:14.785 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 Processor -- start 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 -- start start 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient: init 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient: _add_conns ranks=[4,5,3,0,1,2] 2026-02-22T21:42:14.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc8a0152ab0 0x7fc8a00c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a0152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fc8a006d9d0 0x7fc8a00c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a006d9d0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 0x7fc8a00c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a00c5ff0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a006ed40 con 0x7fc8a00c5ff0 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a006f1d0 con 0x7fc8a0152ab0 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a006de50 con 0x7fc8a006d9d0 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.782+0000 7fc8a6fa0640 10 monclient(hunting): authenticate will time out at 1410.363667s 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 0x7fc8a00c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fc8a0152ab0 0x7fc8a00c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 0x7fc8a00c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:42480/0 (socket says 10.20.193.51:42480) 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 1 -- 10.20.193.51:0/2119564750 learned_addr learned my addr 10.20.193.51:0/2119564750 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): get_auth_request con 0x7fc8a00c5ff0 auth_method 0 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): get_auth_request con 0x7fc8a0152ab0 auth_method 0 2026-02-22T21:42:14.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc89ffff640 1 --2- 10.20.193.51:0/2119564750 >> v2:10.20.193.125:3300/0 conn(0x7fc8a006d9d0 0x7fc8a00c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc89ffff640 10 monclient(hunting): get_auth_request con 0x7fc8a006d9d0 auth_method 0 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc89ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc89ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.783+0000 7fc89ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_done global_id 5092 payload 274 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 -- 10.20.193.51:0/2119564750 >> v2:10.20.193.125:3300/0 conn(0x7fc8a006d9d0 msgr2=0x7fc8a00c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/2119564750 >> v2:10.20.193.125:3300/0 conn(0x7fc8a006d9d0 0x7fc8a00c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 -- 10.20.193.51:0/2119564750 >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 msgr2=0x7fc8a00c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/2119564750 >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 0x7fc8a00c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 10 monclient: _finish_hunting 0 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 monclient: found mon.noname-b 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:14.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:14.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.784+0000 7fc8a4d15640 1 -- 10.20.193.51:0/2119564750 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a006e130 con 0x7fc8a0152ab0 2026-02-22T21:42:14.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.785+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/2119564750 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a0152ab0 0x7fc8a00c5ab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc8a006e060 tx=0x7fc89002ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af2b400dc44a6eeb server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 1 -- 10.20.193.51:0/2119564750 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89003c070 con 0x7fc8a0152ab0 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: dump: 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:14.800 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:14.801 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 5 monclient: authenticate success, global_id 5092 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 1 -- 10.20.193.51:0/2119564750 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89002fb10 con 0x7fc8a0152ab0 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 1 -- 10.20.193.51:0/2119564750 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890037440 con 0x7fc8a0152ab0 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 4 set_mon_vals no callback set 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc89f7fe640 10 monclient: dump: 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:14.802 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 10 monclient: shutdown 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/2119564750 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a0152ab0 msgr2=0x7fc8a00c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/2119564750 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a0152ab0 0x7fc8a00c5ab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc8a006e060 tx=0x7fc89002ee90 comp rx=0 tx=0).stop 2026-02-22T21:42:14.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/2119564750 shutdown_connections 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/2119564750 >> v2:10.20.193.51:3300/0 conn(0x7fc8a00c5ff0 0x7fc8a00c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/2119564750 >> v2:10.20.193.125:3300/0 conn(0x7fc8a006d9d0 0x7fc8a00c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/2119564750 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a0152ab0 0x7fc8a00c5ab0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/2119564750 >> 10.20.193.51:0/2119564750 conn(0x7fc8a00cd7b0 msgr2=0x7fc8a00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/2119564750 shutdown_connections 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.787+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/2119564750 wait complete. 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient: build_initial_monmap 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient: monmap: 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:14.804 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 librados: starting msgr at 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 librados: starting objecter 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 Processor -- start 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 -- start start 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 librados: setting wanted keys 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 librados: calling monclient init 2026-02-22T21:42:14.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient: init 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a006d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc8a00c5ff0 0x7fc8a01f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a00c5ff0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8a0152ab0 0x7fc8a01f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a0152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a006e9d0 con 0x7fc8a0152ab0 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a006e6f0 con 0x7fc8a006d9d0 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0152e90 con 0x7fc8a00c5ff0 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.788+0000 7fc8a6fa0640 10 monclient(hunting): authenticate will time out at 1410.369909s 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:47716/0 (socket says 10.20.193.51:47716) 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 1 -- 10.20.193.51:0/3275613742 learned_addr learned my addr 10.20.193.51:0/3275613742 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): get_auth_request con 0x7fc8a006d9d0 auth_method 0 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8a0152ab0 0x7fc8a01f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc89ffff640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc8a00c5ff0 0x7fc8a01f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): get_auth_request con 0x7fc8a0152ab0 auth_method 0 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc89ffff640 10 monclient(hunting): get_auth_request con 0x7fc8a00c5ff0 auth_method 0 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc89ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc89ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc89ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.789+0000 7fc8a5516640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc89ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 10 monclient(hunting): handle_auth_done global_id 5095 payload 824 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 -- 10.20.193.51:0/3275613742 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc8a00c5ff0 msgr2=0x7fc8a01f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc8a00c5ff0 0x7fc8a01f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 -- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8a0152ab0 msgr2=0x7fc8a01f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8a0152ab0 0x7fc8a01f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 10 monclient: _finish_hunting 0 2026-02-22T21:42:14.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 monclient: found mon.noname-b 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.790+0000 7fc8a4d15640 1 -- 10.20.193.51:0/3275613742 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a01f31b0 con 0x7fc8a006d9d0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.792+0000 7fc8a4d15640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc89003a040 tx=0x7fc8900047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.792+0000 7fc89d7fa640 10 client.?.objecter ms_handle_connect 0x7fc8a006d9d0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.792+0000 7fc89d7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890047070 con 0x7fc8a006d9d0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: dump: 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:14.810 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 5 monclient: authenticate success, global_id 5095 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 10 monclient: _renew_subs 2026-02-22T21:42:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a01f3490 con 0x7fc8a006d9d0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 10 client.5095.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 10 monclient: _renew_subs 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a01f3a50 con 0x7fc8a006d9d0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc8a6fa0640 1 librados: init done 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89002f9d0 con 0x7fc8a006d9d0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89effd640 4 set_mon_vals no callback set 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890051060 con 0x7fc8a006d9d0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.793+0000 7fc89d7fa640 10 monclient: dump: 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:14.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.794+0000 7fc8a6fa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:14.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.794+0000 7fc8a6fa0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:14.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.794+0000 7fc8a6fa0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:14.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.794+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a00c5ab0 con 0x7fc8a006d9d0 2026-02-22T21:42:14.903 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.901+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc890004430 con 0x7fc8a006d9d0 2026-02-22T21:42:14.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.901+0000 7fc89d7fa640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 0x7fc87007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:14.904 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.901+0000 7fc89ffff640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 0x7fc87007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:14.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.901+0000 7fc89ffff640 10 monclient: get_auth_request con 0x7fc870078e00 auth_method 0 2026-02-22T21:42:14.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.902+0000 7fc89ffff640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 0x7fc87007b2c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc8a014c880 tx=0x7fc888009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:14.905 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.902+0000 7fc89d7fa640 10 client.5095.objecter ms_handle_connect 0x7fc870078e00 2026-02-22T21:42:14.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.903+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc8900bc280 con 0x7fc8a006d9d0 2026-02-22T21:42:14.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.903+0000 7fc89d7fa640 10 client.5095.objecter ms_dispatch2 0x7fc8a0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:14.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.903+0000 7fc89d7fa640 3 client.5095.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:14.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.903+0000 7fc89d7fa640 3 client.5095.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:14.906 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.903+0000 7fc89d7fa640 20 client.5095.objecter dump_active .. 0 homeless 2026-02-22T21:42:14.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.909+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc890086b60 con 0x7fc8a006d9d0 2026-02-22T21:42:14.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.909+0000 7fc89d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:14.912 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:14.909+0000 7fc89d7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:15.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.008+0000 7fc8a6fa0640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:15.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.008+0000 7fc8a6fa0640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:15.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.008+0000 7fc8a6fa0640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:15.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.008+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc8a00c86f0 con 0x7fc8a006d9d0 2026-02-22T21:42:15.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.015+0000 7fc89d7fa640 1 -- 10.20.193.51:0/3275613742 <== mon.1 v2:10.20.193.68:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fc89008ae60 con 0x7fc8a006d9d0 2026-02-22T21:42:15.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.015+0000 7fc89d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:15.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.015+0000 7fc89d7fa640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:15.018 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:15.018 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 10 librados: watch_flush enter 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 10 librados: watch_flush exit 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 20 client.5095.objecter shutdown clearing up homeless session... 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 10 client.5095.objecter successfully canceled tick 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 msgr2=0x7fc87007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:15.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 0x7fc87007b2c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc8a014c880 tx=0x7fc888009290 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 10 monclient: shutdown 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 msgr2=0x7fc8a01ef650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc89003a040 tx=0x7fc8900047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 shutdown_connections 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc870078e00 0x7fc87007b2c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8a0152ab0 0x7fc8a01f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc8a00c5ff0 0x7fc8a01f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 --2- 10.20.193.51:0/3275613742 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8a006d9d0 0x7fc8a01ef650 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.016+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 >> 10.20.193.51:0/3275613742 conn(0x7fc8a00cd7b0 msgr2=0x7fc8a00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.017+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 shutdown_connections 2026-02-22T21:42:15.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.017+0000 7fc8a6fa0640 1 -- 10.20.193.51:0/3275613742 wait complete. 2026-02-22T21:42:15.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:15.017+0000 7fc8a6fa0640 1 librados: shutdown 2026-02-22T21:42:18.029 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:18.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.094+0000 7f6d9034d640 10 monclient: get_monmap_and_config 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.094+0000 7f6d9034d640 10 monclient: build_initial_monmap 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.094+0000 7f6d9034d640 10 monclient: monmap: 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:18.096453+0000 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:18.096453+0000 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:18.098 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 Processor -- start 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 -- start start 2026-02-22T21:42:18.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient: init 2026-02-22T21:42:18.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient: _add_conns ranks=[3,5,2,4,0,1] 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6d88150220 0x7f6d880c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d88150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6d88150600 0x7f6d8815c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d88150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d8815c540 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d8815c9b0 con 0x7f6d8815c540 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f6d8815a920 con 0x7f6d88150220 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:18.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6d8815af60 con 0x7f6d88150600 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.095+0000 7f6d9034d640 10 monclient(hunting): authenticate will time out at 1413.676777s 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6d88150220 0x7f6d880c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6d88150220 0x7f6d880c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:38356/0 (socket says 10.20.193.51:38356) 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 1 -- 10.20.193.51:0/3553000282 learned_addr learned my addr 10.20.193.51:0/3553000282 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): get_auth_request con 0x7f6d88150220 auth_method 0 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:18.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.096+0000 7f6d8e0c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 10 monclient(hunting): handle_auth_done global_id 5100 payload 274 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 1 -- 10.20.193.51:0/3553000282 >> v2:10.20.193.125:3300/0 conn(0x7f6d88150600 msgr2=0x7f6d8815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 1 --2- 10.20.193.51:0/3553000282 >> v2:10.20.193.125:3300/0 conn(0x7f6d88150600 0x7f6d8815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 1 -- 10.20.193.51:0/3553000282 >> v1:10.20.193.125:6789/0 conn(0x7f6d8815c540 legacy=0x7f6d8806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 10 monclient: _finish_hunting 0 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 1 monclient: found mon.noname-a 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.097+0000 7f6d8e0c2640 1 -- 10.20.193.51:0/3553000282 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8815b240 con 0x7f6d88150220 2026-02-22T21:42:18.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.098+0000 7f6d8e0c2640 1 --2- 10.20.193.51:0/3553000282 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d88150220 0x7f6d880c73a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6d88074940 tx=0x7f6d7802ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae899a10ddd9f0b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 1 -- 10.20.193.51:0/3553000282 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7803c070 con 0x7f6d88150220 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: dump: 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:18.109 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: _finish_auth 0 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: _check_auth_tickets 2026-02-22T21:42:18.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 1 -- 10.20.193.51:0/3553000282 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d7802fbd0 con 0x7f6d88150220 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d9034d640 5 monclient: authenticate success, global_id 5100 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d9034d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 1 -- 10.20.193.51:0/3553000282 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78037440 con 0x7f6d88150220 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d8d0c0640 10 monclient: dump: 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.111 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d9034d640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.100+0000 7f6d9034d640 4 set_mon_vals no callback set 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 10 monclient: shutdown 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- 10.20.193.51:0/3553000282 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d88150220 msgr2=0x7f6d880c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:18.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 --2- 10.20.193.51:0/3553000282 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d88150220 0x7f6d880c73a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6d88074940 tx=0x7f6d7802ef60 comp rx=0 tx=0).stop 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- 10.20.193.51:0/3553000282 shutdown_connections 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 --2- 10.20.193.51:0/3553000282 >> v2:10.20.193.125:3300/0 conn(0x7f6d88150600 0x7f6d8815c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 --2- 10.20.193.51:0/3553000282 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d88150220 0x7f6d880c73a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- 10.20.193.51:0/3553000282 >> 10.20.193.51:0/3553000282 conn(0x7f6d880cd7b0 msgr2=0x7f6d880cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- 10.20.193.51:0/3553000282 shutdown_connections 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- 10.20.193.51:0/3553000282 wait complete. 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 10 monclient: build_initial_monmap 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 10 monclient: monmap: 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:18.113 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 librados: starting msgr at 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 librados: starting objecter 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 Processor -- start 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 -- start start 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 librados: setting wanted keys 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 1 librados: calling monclient init 2026-02-22T21:42:18.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.101+0000 7f6d9034d640 10 monclient: init 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d88150220 0x7f6d88158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d88150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d88150600 0x7f6d88158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d88150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d8815c540 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8815bae0 con 0x7f6d8815c540 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8815b800 con 0x7f6d88150600 2026-02-22T21:42:18.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6d880703b0 con 0x7f6d88150220 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d9034d640 10 monclient(hunting): authenticate will time out at 1413.683455s 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:38358/0 (socket says 10.20.193.51:38358) 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 1 -- 10.20.193.51:0/1085712084 learned_addr learned my addr 10.20.193.51:0/1085712084 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 10 monclient(hunting): get_auth_request con 0x7f6d8815c540 auth_method 0 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.102+0000 7f6d8e8c3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.103+0000 7f6d8e8c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:18.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.103+0000 7f6d8e8c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.103+0000 7f6d8e8c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 10 monclient(hunting): handle_auth_done global_id 5103 payload 824 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 -- 10.20.193.51:0/1085712084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d88150220 msgr2=0x7f6d88158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d88150220 0x7f6d88158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 -- 10.20.193.51:0/1085712084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d88150600 msgr2=0x7f6d88158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d88150600 0x7f6d88158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 10 monclient: _finish_hunting 0 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 monclient: found mon.noname-a 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.104+0000 7f6d8e8c3640 1 -- 10.20.193.51:0/1085712084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8814d3f0 con 0x7f6d8815c540 2026-02-22T21:42:18.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.105+0000 7f6d8e8c3640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400b990 tx=0x7f6d8400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:18.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.105+0000 7f6d6effd640 10 client.?.objecter ms_handle_connect 0x7f6d8815c540 2026-02-22T21:42:18.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.105+0000 7f6d6effd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:18.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400c790 con 0x7f6d8815c540 2026-02-22T21:42:18.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: dump: 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:18.120 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: _finish_auth 0 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d84010070 con 0x7f6d8815c540 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400cbb0 con 0x7f6d8815c540 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d6effd640 10 monclient: dump: 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:18.121 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d8c8bf640 4 set_mon_vals no callback set 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 5 monclient: authenticate success, global_id 5103 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 10 monclient: _renew_subs 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d881558f0 con 0x7f6d8815c540 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 10 client.5103.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 10 monclient: _renew_subs 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d88157a70 con 0x7f6d8815c540 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.107+0000 7f6d9034d640 1 librados: init done 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.108+0000 7f6d9034d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.108+0000 7f6d9034d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.108+0000 7f6d9034d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.108+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d8815c000 con 0x7f6d8815c540 2026-02-22T21:42:18.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6d84004070 con 0x7f6d8815c540 2026-02-22T21:42:18.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d6effd640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 0x7f6d5807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:18.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d8e0c2640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 0x7f6d5807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:18.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d8e0c2640 10 monclient: get_auth_request con 0x7f6d58078e00 auth_method 0 2026-02-22T21:42:18.219 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d8e0c2640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 0x7f6d5807b2c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d880c65c0 tx=0x7f6d78031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:18.220 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.216+0000 7f6d6effd640 10 client.5103.objecter ms_handle_connect 0x7f6d58078e00 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f6d8400cd50 con 0x7f6d8815c540 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 10 client.5103.objecter ms_dispatch2 0x7f6d88000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 3 client.5103.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 3 client.5103.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 20 client.5103.objecter dump_active .. 0 homeless 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6d840622f0 con 0x7f6d8815c540 2026-02-22T21:42:18.228 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:18.229 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.225+0000 7f6d6effd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:18.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.325+0000 7f6d9034d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:18.328 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.325+0000 7f6d9034d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:18.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.325+0000 7f6d9034d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:18.329 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.325+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6d88074da0 con 0x7f6d8815c540 2026-02-22T21:42:18.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.328+0000 7f6d6effd640 1 -- 10.20.193.51:0/1085712084 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f6d84061bf0 con 0x7f6d8815c540 2026-02-22T21:42:18.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.328+0000 7f6d6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:18.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.328+0000 7f6d6effd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:18.331 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:18.331 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 10 librados: watch_flush enter 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 10 librados: watch_flush exit 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 20 client.5103.objecter shutdown clearing up homeless session... 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 10 client.5103.objecter successfully canceled tick 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 msgr2=0x7f6d5807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 0x7f6d5807b2c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d880c65c0 tx=0x7f6d78031000 comp rx=0 tx=0).stop 2026-02-22T21:42:18.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 10 monclient: shutdown 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 msgr2=0x7f6d8814c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400b990 tx=0x7f6d8400be60 comp rx=0 tx=0).stop 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 shutdown_connections 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6d58078e00 0x7f6d5807b2c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6d8815c540 0x7f6d8814c3c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6d88150600 0x7f6d88158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 --2- 10.20.193.51:0/1085712084 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6d88150220 0x7f6d88158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 >> 10.20.193.51:0/1085712084 conn(0x7f6d880cd7b0 msgr2=0x7f6d880c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 shutdown_connections 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 -- 10.20.193.51:0/1085712084 wait complete. 2026-02-22T21:42:18.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:18.329+0000 7f6d9034d640 1 librados: shutdown 2026-02-22T21:42:21.341 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:21.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.407+0000 7f2ff5cf3640 10 monclient: get_monmap_and_config 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.407+0000 7f2ff5cf3640 10 monclient: build_initial_monmap 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.407+0000 7f2ff5cf3640 10 monclient: monmap: 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:21.409506+0000 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:21.409506+0000 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:21.411 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 Processor -- start 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 -- start start 2026-02-22T21:42:21.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient: init 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient: _add_conns ranks=[5,0,2,1,3,4] 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f2ff0150220 0x7f2ff00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff0150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff0150600 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ff015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff015a920 con 0x7f2ff0150600 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff015af60 con 0x7f2ff015c710 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:21.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff015b240 con 0x7f2ff0150220 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.408+0000 7f2ff5cf3640 10 monclient(hunting): authenticate will time out at 1416.989812s 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2feeffd640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f2ff0150600 0x7f2ff015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:47720/0 (socket says 10.20.193.51:47720) 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2feeffd640 1 -- 10.20.193.51:0/2016849089 learned_addr learned my addr 10.20.193.51:0/2016849089 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 1 --2- 10.20.193.51:0/2016849089 >> v2:10.20.193.125:3300/0 conn(0x7f2ff0150220 0x7f2ff00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request con 0x7f2ff0150220 auth_method 0 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:21.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.409+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 1 -- 10.20.193.51:0/2016849089 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837888979 0 0) 0x7f2ff015a920 con 0x7f2ff0150600 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 10 monclient(hunting): _negotiate 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 10 monclient(hunting): my global_id is 5106 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.410+0000 7f2fee7fc640 1 -- 10.20.193.51:0/2016849089 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fd4003620 con 0x7f2ff0150600 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_done global_id 5063 payload 274 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 1 -- 10.20.193.51:0/2016849089 >> v1:10.20.193.125:6789/0 conn(0x7f2ff015c710 legacy=0x7f2ff006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 1 -- 10.20.193.51:0/2016849089 >> v1:10.20.193.51:6789/0 conn(0x7f2ff0150600 legacy=0x7f2ff015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 1 monclient: found mon.noname-c 2026-02-22T21:42:21.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:21.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:21.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.411+0000 7f2fef7fe640 1 -- 10.20.193.51:0/2016849089 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff015b520 con 0x7f2ff0150220 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.413+0000 7f2fef7fe640 1 --2- 10.20.193.51:0/2016849089 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff00c73a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098a0 tx=0x7f2fdc02ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=641bd11b2c0cf1bc server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 1 -- 10.20.193.51:0/2016849089 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc03c070 con 0x7f2ff0150220 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: dump: 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:21.423 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 1 -- 10.20.193.51:0/2016849089 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc02fc10 con 0x7f2ff0150220 2026-02-22T21:42:21.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 5 monclient: authenticate success, global_id 5063 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 1 -- 10.20.193.51:0/2016849089 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc037440 con 0x7f2ff0150220 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 4 set_mon_vals no callback set 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2fee7fc640 10 monclient: dump: 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.425 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 10 monclient: shutdown 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2016849089 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 msgr2=0x7f2ff00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2016849089 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff00c73a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098a0 tx=0x7f2fdc02ef00 comp rx=0 tx=0).stop 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2016849089 shutdown_connections 2026-02-22T21:42:21.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2016849089 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff00c73a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2016849089 >> 10.20.193.51:0/2016849089 conn(0x7f2ff00cd7b0 msgr2=0x7f2ff00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2016849089 shutdown_connections 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.415+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2016849089 wait complete. 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 10 monclient: build_initial_monmap 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 10 monclient: monmap: 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:21.427 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 librados: starting msgr at 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 librados: starting objecter 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 Processor -- start 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 -- start start 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 librados: setting wanted keys 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 1 librados: calling monclient init 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.416+0000 7f2ff5cf3640 10 monclient: init 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff006ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:21.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff0150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2ff015c710 0x7f2ff0150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff00703b0 con 0x7f2ff0150600 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff015bae0 con 0x7f2ff015c710 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff0070530 con 0x7f2ff0150220 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:21.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2ff5cf3640 10 monclient(hunting): authenticate will time out at 1416.998125s 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff006ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:38362/0 (socket says 10.20.193.51:38362) 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 1 -- 10.20.193.51:0/2960358646 learned_addr learned my addr 10.20.193.51:0/2960358646 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): get_auth_request con 0x7f2ff0150600 auth_method 0 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feffff640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2ff015c710 0x7f2ff0150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request con 0x7f2ff0150220 auth_method 0 2026-02-22T21:42:21.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feffff640 10 monclient(hunting): get_auth_request con 0x7f2ff015c710 auth_method 0 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2feffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:21.431 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.417+0000 7f2fef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.418+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.418+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.418+0000 7f2feffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 10 monclient(hunting): handle_auth_done global_id 5109 payload 824 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 -- 10.20.193.51:0/2960358646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 msgr2=0x7f2ff006ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff006ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 -- 10.20.193.51:0/2960358646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2ff015c710 msgr2=0x7f2ff0150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2ff015c710 0x7f2ff0150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 monclient: found mon.noname-a 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:21.432 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:21.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.419+0000 7f2feeffd640 1 -- 10.20.193.51:0/2960358646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff006f0f0 con 0x7f2ff0150600 2026-02-22T21:42:21.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.420+0000 7f2feeffd640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c520 tx=0x7f2fe000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:21.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.420+0000 7f2fcbfff640 10 client.?.objecter ms_handle_connect 0x7f2ff0150600 2026-02-22T21:42:21.433 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.420+0000 7f2fcbfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:21.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0017070 con 0x7f2ff0150600 2026-02-22T21:42:21.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:21.435 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: dump: 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:21.436 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: _finish_auth 0 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe0005de0 con 0x7f2ff0150600 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 5 monclient: authenticate success, global_id 5109 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 10 monclient: _renew_subs 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff01f2140 con 0x7f2ff0150600 2026-02-22T21:42:21.437 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0007270 con 0x7f2ff0150600 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fcbfff640 10 monclient: dump: 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:21.438 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2fedffb640 4 set_mon_vals no callback set 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 10 client.5109.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 10 monclient: _renew_subs 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff01f2700 con 0x7f2ff0150600 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.421+0000 7f2ff5cf3640 1 librados: init done 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.422+0000 7f2ff5cf3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.422+0000 7f2ff5cf3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.422+0000 7f2ff5cf3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:21.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.422+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff00c73a0 con 0x7f2ff0150600 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.433+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f2fe0007410 con 0x7f2ff0150600 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.433+0000 7f2fcbfff640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 0x7f2fc007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.434+0000 7f2fef7fe640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 0x7f2fc007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.434+0000 7f2fef7fe640 10 monclient: get_auth_request con 0x7f2fc0078e00 auth_method 0 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.434+0000 7f2fef7fe640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 0x7f2fc007b2c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc039000 tx=0x7f2fdc033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.434+0000 7f2fcbfff640 10 client.5109.objecter ms_handle_connect 0x7f2fc0078e00 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.435+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f2fe0007730 con 0x7f2ff0150600 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.435+0000 7f2fcbfff640 10 client.5109.objecter ms_dispatch2 0x7f2ff0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:21.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.435+0000 7f2fcbfff640 3 client.5109.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:21.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.435+0000 7f2fcbfff640 3 client.5109.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:21.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.435+0000 7f2fcbfff640 20 client.5109.objecter dump_active .. 0 homeless 2026-02-22T21:42:21.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.538+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f2fe00613e0 con 0x7f2ff0150600 2026-02-22T21:42:21.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.538+0000 7f2fcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:21.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.538+0000 7f2fcbfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:21.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.640+0000 7f2ff5cf3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:21.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.640+0000 7f2ff5cf3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:21.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.640+0000 7f2ff5cf3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:21.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.640+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2ff0074da0 con 0x7f2ff0150600 2026-02-22T21:42:21.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.643+0000 7f2fcbfff640 1 -- 10.20.193.51:0/2960358646 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f2fe00656e0 con 0x7f2ff0150600 2026-02-22T21:42:21.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.643+0000 7f2fcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:21.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.643+0000 7f2fcbfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:21.646 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:21.646 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 10 librados: watch_flush enter 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 10 librados: watch_flush exit 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 20 client.5109.objecter shutdown clearing up homeless session... 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 10 client.5109.objecter successfully canceled tick 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 msgr2=0x7f2fc007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 0x7f2fc007b2c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc039000 tx=0x7f2fdc033000 comp rx=0 tx=0).stop 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 10 monclient: shutdown 2026-02-22T21:42:21.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 msgr2=0x7f2ff014c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c520 tx=0x7f2fe000c9f0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 shutdown_connections 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f2fc0078e00 0x7f2fc007b2c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f2ff015c710 0x7f2ff0150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f2ff0150600 0x7f2ff014c800 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 --2- 10.20.193.51:0/2960358646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f2ff0150220 0x7f2ff006ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 >> 10.20.193.51:0/2960358646 conn(0x7f2ff00cd7b0 msgr2=0x7f2ff00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 shutdown_connections 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 -- 10.20.193.51:0/2960358646 wait complete. 2026-02-22T21:42:21.649 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:21.644+0000 7f2ff5cf3640 1 librados: shutdown 2026-02-22T21:42:21.727 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:52.881 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.729 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:52.882 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.737 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:06:52.891 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.754 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:52.870 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.756 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:52.872 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.756 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:52.908 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.758 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:52.910 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.764 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:06:52.880 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:21.766 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:06:52.918 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:24.656 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:24.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 10 monclient: get_monmap_and_config 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 10 monclient: build_initial_monmap 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 10 monclient: monmap: 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:24.722795+0000 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:24.722795+0000 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:24.724 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:24.725 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:24.725 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:24.725 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:24.725 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 1 Processor -- start 2026-02-22T21:42:24.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 1 -- start start 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.721+0000 7fa937797640 10 monclient: init 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient: _add_conns ranks=[5,4,3,2,0,1] 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa9301526a0 0x7fa9300733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): picked mon.noname-c con 0x7fa9301526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa930152a80 0x7fa9300cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): picked mon.noname-b con 0x7fa930152a80 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): picked mon.noname-a con 0x7fa9300cdb20 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:24.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa9300cca00 con 0x7fa9300cdb20 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fa930074c50 con 0x7fa930152a80 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fa930073d60 con 0x7fa9301526a0 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa937797640 10 monclient(hunting): authenticate will time out at 1420.303201s 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa934d0b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa930152a80 0x7fa9300cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa93550c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa9301526a0 0x7fa9300733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:38370/0 (socket says 10.20.193.51:38370) 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 1 -- 10.20.193.51:0/1531086786 learned_addr learned my addr 10.20.193.51:0/1531086786 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 10 monclient(hunting): get_auth_request con 0x7fa9300cdb20 auth_method 0 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa935d0d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa934d0b640 10 monclient(hunting): get_auth_request con 0x7fa930152a80 auth_method 0 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa934d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa934d0b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa934d0b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.722+0000 7fa93550c640 10 monclient(hunting): get_auth_request con 0x7fa9301526a0 auth_method 0 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.732 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.723+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 10 monclient(hunting): handle_auth_done global_id 5072 payload 274 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 -- 10.20.193.51:0/1531086786 >> v2:10.20.193.68:3300/0 conn(0x7fa930152a80 msgr2=0x7fa9300cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 --2- 10.20.193.51:0/1531086786 >> v2:10.20.193.68:3300/0 conn(0x7fa930152a80 0x7fa9300cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 -- 10.20.193.51:0/1531086786 >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 msgr2=0x7fa93015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 --2- 10.20.193.51:0/1531086786 >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 monclient: found mon.noname-c 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa93550c640 1 -- 10.20.193.51:0/1531086786 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa930074040 con 0x7fa9301526a0 2026-02-22T21:42:24.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.724+0000 7fa935d0d640 1 --2- 10.20.193.51:0/1531086786 >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:42:24.734 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.725+0000 7fa93550c640 1 --2- 10.20.193.51:0/1531086786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9301526a0 0x7fa9300733d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa93015f410 tx=0x7fa92002ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a0d9157594e22f2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:24.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 1 -- 10.20.193.51:0/1531086786 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92003c070 con 0x7fa9301526a0 2026-02-22T21:42:24.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:24.741 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: dump: 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:24.742 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: _finish_auth 0 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 1 -- 10.20.193.51:0/1531086786 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92002fb00 con 0x7fa9301526a0 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 5 monclient: authenticate success, global_id 5072 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 1 -- 10.20.193.51:0/1531086786 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920037440 con 0x7fa9301526a0 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:24.743 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 4 set_mon_vals no callback set 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa91ffff640 10 monclient: dump: 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.744 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 10 monclient: shutdown 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 -- 10.20.193.51:0/1531086786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9301526a0 msgr2=0x7fa9300733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 --2- 10.20.193.51:0/1531086786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9301526a0 0x7fa9300733d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa93015f410 tx=0x7fa92002ee40 comp rx=0 tx=0).stop 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 -- 10.20.193.51:0/1531086786 shutdown_connections 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 --2- 10.20.193.51:0/1531086786 >> v2:10.20.193.51:3300/0 conn(0x7fa9300cdb20 0x7fa93015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 --2- 10.20.193.51:0/1531086786 >> v2:10.20.193.68:3300/0 conn(0x7fa930152a80 0x7fa9300cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 --2- 10.20.193.51:0/1531086786 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9301526a0 0x7fa9300733d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 -- 10.20.193.51:0/1531086786 >> 10.20.193.51:0/1531086786 conn(0x7fa9300ce350 msgr2=0x7fa9300ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 -- 10.20.193.51:0/1531086786 shutdown_connections 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.727+0000 7fa937797640 1 -- 10.20.193.51:0/1531086786 wait complete. 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient: build_initial_monmap 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient: monmap: 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:24.746 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 librados: starting msgr at 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 librados: starting objecter 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 Processor -- start 2026-02-22T21:42:24.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 -- start start 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 librados: setting wanted keys 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 librados: calling monclient init 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient: init 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 0x7fa93015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): picked mon.noname-c con 0x7fa9300cdb20 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa9301526a0 0x7fa930149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): picked mon.noname-b con 0x7fa9301526a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 0x7fa93014a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.748 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): picked mon.noname-a con 0x7fa930152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa9300748e0 con 0x7fa930152a80 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa930074600 con 0x7fa9301526a0 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa930073910 con 0x7fa9300cdb20 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.728+0000 7fa937797640 10 monclient(hunting): authenticate will time out at 1420.309819s 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 0x7fa93014a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa9301526a0 0x7fa930149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.749 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 0x7fa93014a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:38372/0 (socket says 10.20.193.51:38372) 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 1 -- 10.20.193.51:0/2053366640 learned_addr learned my addr 10.20.193.51:0/2053366640 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): get_auth_request con 0x7fa930152a80 auth_method 0 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 0x7fa93015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): get_auth_request con 0x7fa9300cdb20 auth_method 0 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.750 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): get_auth_request con 0x7fa9301526a0 auth_method 0 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa935d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa93550c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.751 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.729+0000 7fa934d0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 10 monclient(hunting): handle_auth_done global_id 5075 payload 824 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 -- 10.20.193.51:0/2053366640 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa9301526a0 msgr2=0x7fa930149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa9301526a0 0x7fa930149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 -- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 msgr2=0x7fa93014a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 0x7fa93014a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 monclient: found mon.noname-c 2026-02-22T21:42:24.752 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.730+0000 7fa93550c640 1 -- 10.20.193.51:0/2053366640 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93015aac0 con 0x7fa9300cdb20 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.731+0000 7fa93550c640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 0x7fa93015ff70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa920036000 tx=0x7fa9200047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.731+0000 7fa91dffb640 10 client.?.objecter ms_handle_connect 0x7fa9300cdb20 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.731+0000 7fa91dffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920046020 con 0x7fa9300cdb20 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: dump: 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:24.753 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:24.754 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa920042430 con 0x7fa9300cdb20 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 5 monclient: authenticate success, global_id 5075 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 10 monclient: _renew_subs 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93015ada0 con 0x7fa9300cdb20 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92003c040 con 0x7fa9300cdb20 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:24.755 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91f7fe640 4 set_mon_vals no callback set 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa91dffb640 10 monclient: dump: 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:24.756 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 10 client.5075.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 10 monclient: _renew_subs 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93015b360 con 0x7fa9300cdb20 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.733+0000 7fa937797640 1 librados: init done 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.734+0000 7fa937797640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.734+0000 7fa937797640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.734+0000 7fa937797640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.734+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9300733d0 con 0x7fa9300cdb20 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.745+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa920053070 con 0x7fa9300cdb20 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.746+0000 7fa91dffb640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 0x7fa90007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.746+0000 7fa934d0b640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 0x7fa90007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.746+0000 7fa934d0b640 10 monclient: get_auth_request con 0x7fa900078e50 auth_method 0 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.746+0000 7fa91dffb640 10 client.5075.objecter ms_handle_connect 0x7fa900078e50 2026-02-22T21:42:24.759 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.746+0000 7fa934d0b640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 0x7fa90007b310 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa93014e0c0 tx=0x7fa924009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:24.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.840+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa9200834c0 con 0x7fa9300cdb20 2026-02-22T21:42:24.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.840+0000 7fa91dffb640 10 client.5075.objecter ms_dispatch2 0x7fa930000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:24.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.840+0000 7fa91dffb640 3 client.5075.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:24.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.840+0000 7fa91dffb640 3 client.5075.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:24.843 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.840+0000 7fa91dffb640 20 client.5075.objecter dump_active .. 0 homeless 2026-02-22T21:42:24.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.847+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa920050020 con 0x7fa9300cdb20 2026-02-22T21:42:24.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.847+0000 7fa91dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:24.849 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.847+0000 7fa91dffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.946+0000 7fa937797640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.946+0000 7fa937797640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.946+0000 7fa937797640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:24.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.946+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa9300cb1f0 con 0x7fa9300cdb20 2026-02-22T21:42:24.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.949+0000 7fa91dffb640 1 -- 10.20.193.51:0/2053366640 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa9300cb1f0 con 0x7fa9300cdb20 2026-02-22T21:42:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.949+0000 7fa91dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:24.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.949+0000 7fa91dffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:24.952 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:24.952 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 10 librados: watch_flush enter 2026-02-22T21:42:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 10 librados: watch_flush exit 2026-02-22T21:42:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 20 client.5075.objecter shutdown clearing up homeless session... 2026-02-22T21:42:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 10 client.5075.objecter successfully canceled tick 2026-02-22T21:42:24.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 msgr2=0x7fa90007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 0x7fa90007b310 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa93014e0c0 tx=0x7fa924009290 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 10 monclient: shutdown 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 msgr2=0x7fa93015ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 0x7fa93015ff70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa920036000 tx=0x7fa9200047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 shutdown_connections 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa900078e50 0x7fa90007b310 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa930152a80 0x7fa93014a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa9301526a0 0x7fa930149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 --2- 10.20.193.51:0/2053366640 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa9300cdb20 0x7fa93015ff70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:24.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 >> 10.20.193.51:0/2053366640 conn(0x7fa9300ce350 msgr2=0x7fa9300ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 shutdown_connections 2026-02-22T21:42:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 -- 10.20.193.51:0/2053366640 wait complete. 2026-02-22T21:42:24.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:24.950+0000 7fa937797640 1 librados: shutdown 2026-02-22T21:42:27.962 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:28.030 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.028+0000 7ff7259b3640 10 monclient: get_monmap_and_config 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.028+0000 7ff7259b3640 10 monclient: build_initial_monmap 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.028+0000 7ff7259b3640 10 monclient: monmap: 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:28.029355+0000 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:28.029355+0000 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:28.031 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 Processor -- start 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 -- start start 2026-02-22T21:42:28.032 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient: init 2026-02-22T21:42:28.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:28.040 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient: _add_conns ranks=[5,3,2,4,1,0] 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff720152ab0 0x7ff7200c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-c con 0x7ff720152ab0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff72006d9d0 0x7ff7200c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-a con 0x7ff72006d9d0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7200c5ff0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff72006f1d0 con 0x7ff7200c5ff0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff72006ed40 con 0x7ff72006d9d0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ff72006de50 con 0x7ff720152ab0 2026-02-22T21:42:28.041 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.029+0000 7ff7259b3640 10 monclient(hunting): authenticate will time out at 1423.609770s 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff72006d9d0 0x7ff7200c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71effd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff720152ab0 0x7ff7200c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff72006d9d0 0x7ff7200c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36062/0 (socket says 10.20.193.51:36062) 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 1 -- 10.20.193.51:0/3546487403 learned_addr learned my addr 10.20.193.51:0/3546487403 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): get_auth_request con 0x7ff72006d9d0 auth_method 0 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:28.042 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71effd640 10 monclient(hunting): get_auth_request con 0x7ff720152ab0 auth_method 0 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.030+0000 7ff71effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_done global_id 5118 payload 274 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 1 -- 10.20.193.51:0/3546487403 >> v2:10.20.193.125:3300/0 conn(0x7ff720152ab0 msgr2=0x7ff7200c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 1 --2- 10.20.193.51:0/3546487403 >> v2:10.20.193.125:3300/0 conn(0x7ff720152ab0 0x7ff7200c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 1 -- 10.20.193.51:0/3546487403 >> v1:10.20.193.125:6789/0 conn(0x7ff7200c5ff0 legacy=0x7ff72006fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.043 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 1 monclient: found mon.noname-a 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.031+0000 7ff71e7fc640 1 -- 10.20.193.51:0/3546487403 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72006e130 con 0x7ff72006d9d0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.032+0000 7ff71e7fc640 1 --2- 10.20.193.51:0/3546487403 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff72006d9d0 0x7ff7200c8e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff72006eec0 tx=0x7ff70802ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ceaf1ce95a93f57 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 1 -- 10.20.193.51:0/3546487403 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70803c070 con 0x7ff72006d9d0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: dump: 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:28.044 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:28.045 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 1 -- 10.20.193.51:0/3546487403 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff70802fc60 con 0x7ff72006d9d0 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 5 monclient: authenticate success, global_id 5118 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 1 -- 10.20.193.51:0/3546487403 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708037660 con 0x7ff72006d9d0 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 4 set_mon_vals no callback set 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff71dffb640 10 monclient: dump: 2026-02-22T21:42:28.046 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 10 monclient: shutdown 2026-02-22T21:42:28.047 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 -- 10.20.193.51:0/3546487403 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff72006d9d0 msgr2=0x7ff7200c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3546487403 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff72006d9d0 0x7ff7200c8e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff72006eec0 tx=0x7ff70802ef90 comp rx=0 tx=0).stop 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 -- 10.20.193.51:0/3546487403 shutdown_connections 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3546487403 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff72006d9d0 0x7ff7200c8e90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3546487403 >> v2:10.20.193.125:3300/0 conn(0x7ff720152ab0 0x7ff7200c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 -- 10.20.193.51:0/3546487403 >> 10.20.193.51:0/3546487403 conn(0x7ff7200cd7b0 msgr2=0x7ff7200cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 -- 10.20.193.51:0/3546487403 shutdown_connections 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.034+0000 7ff7259b3640 1 -- 10.20.193.51:0/3546487403 wait complete. 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient: build_initial_monmap 2026-02-22T21:42:28.048 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient: monmap: 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 librados: starting msgr at 2026-02-22T21:42:28.049 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 librados: starting objecter 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 Processor -- start 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 -- start start 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 librados: setting wanted keys 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 librados: calling monclient init 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient: init 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff72006d9d0 0x7ff7201487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-b con 0x7ff72006d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7200c5ff0 0x7ff7201f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-c con 0x7ff7200c5ff0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.050 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): picked mon.noname-a con 0x7ff720152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff72006e9d0 con 0x7ff720152ab0 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff72006e6f0 con 0x7ff72006d9d0 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff720152e90 con 0x7ff7200c5ff0 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.035+0000 7ff7259b3640 10 monclient(hunting): authenticate will time out at 1423.615863s 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36066/0 (socket says 10.20.193.51:36066) 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 1 -- 10.20.193.51:0/3778710246 learned_addr learned my addr 10.20.193.51:0/3778710246 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:28.051 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): get_auth_request con 0x7ff720152ab0 auth_method 0 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff72006d9d0 0x7ff7201487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7200c5ff0 0x7ff7201f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): get_auth_request con 0x7ff7200c5ff0 auth_method 0 2026-02-22T21:42:28.052 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): get_auth_request con 0x7ff72006d9d0 auth_method 0 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:28.053 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.036+0000 7ff71effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 10 monclient(hunting): handle_auth_done global_id 5121 payload 824 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 -- 10.20.193.51:0/3778710246 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7200c5ff0 msgr2=0x7ff7201f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7200c5ff0 0x7ff7201f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 -- 10.20.193.51:0/3778710246 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff72006d9d0 msgr2=0x7ff7201487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff72006d9d0 0x7ff7201487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.037+0000 7ff71f7fe640 1 -- 10.20.193.51:0/3778710246 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff720149350 con 0x7ff720152ab0 2026-02-22T21:42:28.054 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.038+0000 7ff71f7fe640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cc50 tx=0x7ff71000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:28.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.038+0000 7ff6f7fff640 10 client.?.objecter ms_handle_connect 0x7ff720152ab0 2026-02-22T21:42:28.055 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.038+0000 7ff6f7fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710003a70 con 0x7ff720152ab0 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: dump: 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.058 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff710003c10 con 0x7ff720152ab0 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7100050d0 con 0x7ff720152ab0 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 5 monclient: authenticate success, global_id 5121 2026-02-22T21:42:28.059 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _renew_subs 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7201f7f00 con 0x7ff720152ab0 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff71d7fa640 4 set_mon_vals no callback set 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff6f7fff640 10 monclient: dump: 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:28.060 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 client.5121.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _renew_subs 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7201f7660 con 0x7ff720152ab0 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 1 librados: init done 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.040+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7200c5ab0 con 0x7ff720152ab0 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.051+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff710005270 con 0x7ff720152ab0 2026-02-22T21:42:28.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.052+0000 7ff6f7fff640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 0x7ff6f007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.052+0000 7ff71effd640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 0x7ff6f007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.052+0000 7ff71effd640 10 monclient: get_auth_request con 0x7ff6f0078e00 auth_method 0 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.052+0000 7ff71effd640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 0x7ff6f007b2c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff714009ba0 tx=0x7ff714009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.052+0000 7ff6f7fff640 10 client.5121.objecter ms_handle_connect 0x7ff6f0078e00 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.053+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff71005e740 con 0x7ff720152ab0 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.053+0000 7ff6f7fff640 10 client.5121.objecter ms_dispatch2 0x7ff720000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.053+0000 7ff6f7fff640 3 client.5121.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.053+0000 7ff6f7fff640 3 client.5121.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:28.062 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.053+0000 7ff6f7fff640 20 client.5121.objecter dump_active .. 0 homeless 2026-02-22T21:42:28.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.151+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7ff710062700 con 0x7ff720152ab0 2026-02-22T21:42:28.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.151+0000 7ff6f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:28.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.151+0000 7ff6f7fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:28.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.254+0000 7ff7259b3640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:28.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.254+0000 7ff7259b3640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:28.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.254+0000 7ff7259b3640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:28.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.254+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff7200c6890 con 0x7ff720152ab0 2026-02-22T21:42:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.257+0000 7ff6f7fff640 1 -- 10.20.193.51:0/3778710246 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7ff710066a00 con 0x7ff720152ab0 2026-02-22T21:42:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.257+0000 7ff6f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:28.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.257+0000 7ff6f7fff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:28.258 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:28.258 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 10 librados: watch_flush enter 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 10 librados: watch_flush exit 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 20 client.5121.objecter shutdown clearing up homeless session... 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 10 client.5121.objecter successfully canceled tick 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 msgr2=0x7ff6f007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 0x7ff6f007b2c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff714009ba0 tx=0x7ff714009290 comp rx=0 tx=0).stop 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 10 monclient: shutdown 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 msgr2=0x7ff7201f4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cc50 tx=0x7ff71000a430 comp rx=0 tx=0).stop 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 shutdown_connections 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff6f0078e00 0x7ff6f007b2c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff720152ab0 0x7ff7201f4650 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff7200c5ff0 0x7ff7201f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 --2- 10.20.193.51:0/3778710246 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff72006d9d0 0x7ff7201487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 >> 10.20.193.51:0/3778710246 conn(0x7ff7200cd7b0 msgr2=0x7ff72014ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 shutdown_connections 2026-02-22T21:42:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 -- 10.20.193.51:0/3778710246 wait complete. 2026-02-22T21:42:28.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:28.258+0000 7ff7259b3640 1 librados: shutdown 2026-02-22T21:42:31.268 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:31.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 10 monclient: get_monmap_and_config 2026-02-22T21:42:31.336 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 10 monclient: build_initial_monmap 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 10 monclient: monmap: 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:31.336044+0000 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:31.336044+0000 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:31.337 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 1 Processor -- start 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 1 -- start start 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.335+0000 7fa5716bd640 10 monclient: init 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient: _add_conns ranks=[1,5,4,2,3,0] 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fa56c150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa56c15c710 0x7fa56c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c15c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 0x7fa56c06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56c15a920 con 0x7fa56c150220 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fa56c15af60 con 0x7fa56c150600 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fa56c15b240 con 0x7fa56c15c710 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa5716bd640 10 monclient(hunting): authenticate will time out at 1426.916463s 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa56b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 0x7fa56c06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.338 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.336+0000 7fa56a7fc640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fa56c15c710 0x7fa56c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 0x7fa56c06fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:37110/0 (socket says 10.20.193.51:37110) 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 1 -- 10.20.193.51:0/3357962874 learned_addr learned my addr 10.20.193.51:0/3357962874 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request con 0x7fa56c150600 auth_method 0 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request con 0x7fa56c15c710 auth_method 0 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:31.339 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.337+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_done global_id 5087 payload 274 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 1 -- 10.20.193.51:0/3357962874 >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 msgr2=0x7fa56c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 1 --2- 10.20.193.51:0/3357962874 >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 0x7fa56c06fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 1 -- 10.20.193.51:0/3357962874 >> v1:10.20.193.68:6789/0 conn(0x7fa56c150220 legacy=0x7fa56c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 1 monclient: found mon.noname-c 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.338+0000 7fa56a7fc640 1 -- 10.20.193.51:0/3357962874 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c15b520 con 0x7fa56c15c710 2026-02-22T21:42:31.340 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.339+0000 7fa56a7fc640 1 --2- 10.20.193.51:0/3357962874 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c15c710 0x7fa56c0c73a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098a0 tx=0x7fa55402eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3f4025861d75b2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:31.342 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 1 -- 10.20.193.51:0/3357962874 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55403c070 con 0x7fa56c15c710 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: dump: 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 1 -- 10.20.193.51:0/3357962874 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55402fb80 con 0x7fa56c15c710 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 1 -- 10.20.193.51:0/3357962874 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554037440 con 0x7fa56c15c710 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 5 monclient: authenticate success, global_id 5087 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa569ffb640 10 monclient: dump: 2026-02-22T21:42:31.343 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 4 set_mon_vals no callback set 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 10 monclient: shutdown 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 -- 10.20.193.51:0/3357962874 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c15c710 msgr2=0x7fa56c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 --2- 10.20.193.51:0/3357962874 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c15c710 0x7fa56c0c73a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098a0 tx=0x7fa55402eeb0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 -- 10.20.193.51:0/3357962874 shutdown_connections 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 --2- 10.20.193.51:0/3357962874 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c15c710 0x7fa56c0c73a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 --2- 10.20.193.51:0/3357962874 >> v2:10.20.193.68:3300/0 conn(0x7fa56c150600 0x7fa56c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 -- 10.20.193.51:0/3357962874 >> 10.20.193.51:0/3357962874 conn(0x7fa56c0cd7b0 msgr2=0x7fa56c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 -- 10.20.193.51:0/3357962874 shutdown_connections 2026-02-22T21:42:31.344 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.341+0000 7fa5716bd640 1 -- 10.20.193.51:0/3357962874 wait complete. 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient: build_initial_monmap 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient: monmap: 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 librados: starting msgr at 2026-02-22T21:42:31.346 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 librados: starting objecter 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 Processor -- start 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 -- start start 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 librados: setting wanted keys 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 librados: calling monclient init 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient: init 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa56c150220 0x7fa56c1f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 0x7fa56c1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): picked mon.noname-a con 0x7fa56c15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c0701e0 con 0x7fa56c15c710 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c15bae0 con 0x7fa56c150220 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c070360 con 0x7fa56c150600 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.342+0000 7fa5716bd640 10 monclient(hunting): authenticate will time out at 1426.922957s 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 0x7fa56c1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa56c150220 0x7fa56c1f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 0x7fa56c1e7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36078/0 (socket says 10.20.193.51:36078) 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 1 -- 10.20.193.51:0/2133846993 learned_addr learned my addr 10.20.193.51:0/2133846993 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:39564/0 (socket says 10.20.193.51:39564) 2026-02-22T21:42:31.347 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request con 0x7fa56c15c710 auth_method 0 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request con 0x7fa56c150600 auth_method 0 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): get_auth_request con 0x7fa56c150220 auth_method 0 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.343+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_done global_id 5090 payload 824 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 -- 10.20.193.51:0/2133846993 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa56c150220 msgr2=0x7fa56c1f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa56c150220 0x7fa56c1f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 -- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 msgr2=0x7fa56c1e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 0x7fa56c1e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:31.348 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 monclient: found mon.noname-c 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.345+0000 7fa56a7fc640 1 -- 10.20.193.51:0/2133846993 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c14c3a0 con 0x7fa56c150600 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa56a7fc640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa554036000 tx=0x7fa5540047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 client.?.objecter ms_handle_connect 0x7fa56c150600 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554046020 con 0x7fa56c150600 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:31.349 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 monclient: dump: 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.346+0000 7fa54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 5 monclient: authenticate success, global_id 5090 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _renew_subs 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56c14c680 con 0x7fa56c150600 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 client.5090.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _renew_subs 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56c14cc40 con 0x7fa56c150600 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 1 librados: init done 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56c06fca0 con 0x7fa56c150600 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55402fac0 con 0x7fa56c150600 2026-02-22T21:42:31.350 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55403c040 con 0x7fa56c150600 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa5697fa640 4 set_mon_vals no callback set 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.347+0000 7fa54b7fe640 10 monclient: dump: 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:31.351 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:31.457 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.455+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa55404b440 con 0x7fa56c150600 2026-02-22T21:42:31.458 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.456+0000 7fa54b7fe640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 0x7fa53807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:31.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.456+0000 7fa56affd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 0x7fa53807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:31.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.456+0000 7fa56affd640 10 monclient: get_auth_request con 0x7fa538078e00 auth_method 0 2026-02-22T21:42:31.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.456+0000 7fa56affd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 0x7fa53807b2c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa560002770 tx=0x7fa560007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:31.459 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.456+0000 7fa54b7fe640 10 client.5090.objecter ms_handle_connect 0x7fa538078e00 2026-02-22T21:42:31.460 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.457+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa5540bbe90 con 0x7fa56c150600 2026-02-22T21:42:31.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.457+0000 7fa54b7fe640 10 client.5090.objecter ms_dispatch2 0x7fa56c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:31.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.457+0000 7fa54b7fe640 3 client.5090.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:31.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.457+0000 7fa54b7fe640 3 client.5090.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:31.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.457+0000 7fa54b7fe640 20 client.5090.objecter dump_active .. 0 homeless 2026-02-22T21:42:31.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.463+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa554050080 con 0x7fa56c150600 2026-02-22T21:42:31.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.463+0000 7fa54b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:31.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.463+0000 7fa54b7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:31.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.563+0000 7fa5716bd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:31.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.563+0000 7fa5716bd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:31.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.563+0000 7fa5716bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:31.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.563+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa56c074da0 con 0x7fa56c150600 2026-02-22T21:42:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.566+0000 7fa54b7fe640 1 -- 10.20.193.51:0/2133846993 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fa554086770 con 0x7fa56c150600 2026-02-22T21:42:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.566+0000 7fa54b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:31.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.566+0000 7fa54b7fe640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:31.568 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:31.568 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:31.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 10 librados: watch_flush enter 2026-02-22T21:42:31.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 10 librados: watch_flush exit 2026-02-22T21:42:31.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 20 client.5090.objecter shutdown clearing up homeless session... 2026-02-22T21:42:31.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 10 client.5090.objecter successfully canceled tick 2026-02-22T21:42:31.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 msgr2=0x7fa53807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 0x7fa53807b2c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa560002770 tx=0x7fa560007ef0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 10 monclient: shutdown 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 msgr2=0x7fa56c14baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa554036000 tx=0x7fa5540047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 shutdown_connections 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa538078e00 0x7fa53807b2c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa56c15c710 0x7fa56c1e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa56c150600 0x7fa56c14baa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 --2- 10.20.193.51:0/2133846993 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa56c150220 0x7fa56c1f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:31.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 >> 10.20.193.51:0/2133846993 conn(0x7fa56c0cd7b0 msgr2=0x7fa56c06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:31.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 shutdown_connections 2026-02-22T21:42:31.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 -- 10.20.193.51:0/2133846993 wait complete. 2026-02-22T21:42:31.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:31.567+0000 7fa5716bd640 1 librados: shutdown 2026-02-22T21:42:34.577 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:34.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 10 monclient: get_monmap_and_config 2026-02-22T21:42:34.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 10 monclient: build_initial_monmap 2026-02-22T21:42:34.643 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 10 monclient: monmap: 2026-02-22T21:42:34.643 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:34.643016+0000 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:34.643016+0000 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:34.644 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 1 Processor -- start 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 1 -- start start 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.642+0000 7f78f85ed640 10 monclient: init 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient: _add_conns ranks=[0,5,1,2,3,4] 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78f0150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f78f015c710 0x7f78f00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-c con 0x7f78f015c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f78f0150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f015a920 con 0x7f78f0150220 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f015af60 con 0x7f78f0150600 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f78f015b240 con 0x7f78f015c710 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f85ed640 10 monclient(hunting): authenticate will time out at 1430.223432s 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f6362640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f78f0150220 0x7f78f015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:59630/0 (socket says 10.20.193.51:59630) 2026-02-22T21:42:34.645 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.643+0000 7f78f6362640 1 -- 10.20.193.51:0/3316236916 learned_addr learned my addr 10.20.193.51:0/3316236916 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 1 --2- 10.20.193.51:0/3316236916 >> v2:10.20.193.125:3300/0 conn(0x7f78f015c710 0x7f78f00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): get_auth_request con 0x7f78f015c710 auth_method 0 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.644+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792954055 0 0) 0x7f78f015af60 con 0x7f78f0150600 2026-02-22T21:42:34.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _negotiate 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): my global_id is 5116 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d8003620 con 0x7f78f0150600 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383207414 0 0) 0x7f78f015a920 con 0x7f78f0150220 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _negotiate 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 10 monclient(hunting): my global_id is 5127 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f015af60 con 0x7f78f0150220 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 10 monclient(hunting): handle_auth_done global_id 5093 payload 274 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 1 -- 10.20.193.51:0/3316236916 >> v1:10.20.193.68:6789/0 conn(0x7f78f0150600 legacy=0x7f78f006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 1 -- 10.20.193.51:0/3316236916 >> v1:10.20.193.51:6789/0 conn(0x7f78f0150220 legacy=0x7f78f015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 10 monclient: _finish_hunting 0 2026-02-22T21:42:34.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 1 monclient: found mon.noname-c 2026-02-22T21:42:34.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:34.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:34.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.645+0000 7f78f5b61640 1 -- 10.20.193.51:0/3316236916 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f015b520 con 0x7f78f015c710 2026-02-22T21:42:34.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.647+0000 7f78f5b61640 1 --2- 10.20.193.51:0/3316236916 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f015c710 0x7f78f00c73a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098d0 tx=0x7f78e402ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a352f7992ab191f2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:34.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.648+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e403c070 con 0x7f78f015c710 2026-02-22T21:42:34.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.648+0000 7f78f5360640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:34.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:34.650 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: dump: 2026-02-22T21:42:34.650 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: _finish_auth 0 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: _check_auth_tickets 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e402fbf0 con 0x7f78f015c710 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 1 -- 10.20.193.51:0/3316236916 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4037440 con 0x7f78f015c710 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 5 monclient: authenticate success, global_id 5093 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 4 set_mon_vals no callback set 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f5360640 10 monclient: dump: 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:34.651 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 10 monclient: shutdown 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 -- 10.20.193.51:0/3316236916 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f015c710 msgr2=0x7f78f00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 --2- 10.20.193.51:0/3316236916 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f015c710 0x7f78f00c73a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098d0 tx=0x7f78e402ef20 comp rx=0 tx=0).stop 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 -- 10.20.193.51:0/3316236916 shutdown_connections 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 --2- 10.20.193.51:0/3316236916 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f015c710 0x7f78f00c73a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 -- 10.20.193.51:0/3316236916 >> 10.20.193.51:0/3316236916 conn(0x7f78f00cd7b0 msgr2=0x7f78f00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 -- 10.20.193.51:0/3316236916 shutdown_connections 2026-02-22T21:42:34.652 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 1 -- 10.20.193.51:0/3316236916 wait complete. 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 10 monclient: build_initial_monmap 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.649+0000 7f78f85ed640 10 monclient: monmap: 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 librados: starting msgr at 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 librados: starting objecter 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 Processor -- start 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 -- start start 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 librados: setting wanted keys 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 librados: calling monclient init 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient: init 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:34.654 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-a con 0x7f78f0150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f0150600 0x7f78f01533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-c con 0x7f78f0150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f78f015c710 0x7f78f014cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): picked mon.noname-b con 0x7f78f015c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00703b0 con 0x7f78f0150220 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f78f015bae0 con 0x7f78f015c710 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0070530 con 0x7f78f0150600 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f85ed640 10 monclient(hunting): authenticate will time out at 1430.230566s 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f6362640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f6362640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:36092/0 (socket says 10.20.193.51:36092) 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.650+0000 7f78f6362640 1 -- 10.20.193.51:0/166612082 learned_addr learned my addr 10.20.193.51:0/166612082 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): get_auth_request con 0x7f78f0150220 auth_method 0 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f0150600 0x7f78f01533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f78f015c710 0x7f78f014cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): get_auth_request con 0x7f78f0150600 auth_method 0 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:34.655 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6362640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): get_auth_request con 0x7f78f015c710 auth_method 0 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f5b61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.651+0000 7f78f6b63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 10 monclient(hunting): handle_auth_done global_id 5130 payload 824 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 -- 10.20.193.51:0/166612082 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f0150600 msgr2=0x7f78f01533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f0150600 0x7f78f01533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 -- 10.20.193.51:0/166612082 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f78f015c710 msgr2=0x7f78f014cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f78f015c710 0x7f78f014cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 10 monclient: _finish_hunting 0 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 monclient: found mon.noname-a 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:34.656 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.652+0000 7f78f6362640 1 -- 10.20.193.51:0/166612082 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f006d120 con 0x7f78f0150220 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.653+0000 7f78f6362640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c700 tx=0x7f78e000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.653+0000 7f78d6ffd640 10 client.?.objecter ms_handle_connect 0x7f78f0150220 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.653+0000 7f78d6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0017070 con 0x7f78f0150220 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: dump: 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:34.657 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e0005bb0 con 0x7f78f0150220 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0007070 con 0x7f78f0150220 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78d6ffd640 10 monclient: dump: 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f4b5f640 4 set_mon_vals no callback set 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 5 monclient: authenticate success, global_id 5130 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 10 monclient: _renew_subs 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f006d400 con 0x7f78f0150220 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 10 client.5130.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 10 monclient: _renew_subs 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:34.658 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f006d9c0 con 0x7f78f0150220 2026-02-22T21:42:34.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.655+0000 7f78f85ed640 1 librados: init done 2026-02-22T21:42:34.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.656+0000 7f78f85ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:34.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.656+0000 7f78f85ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:34.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.656+0000 7f78f85ed640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:34.659 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.656+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f00c73a0 con 0x7f78f0150220 2026-02-22T21:42:34.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.667+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f78e0007210 con 0x7f78f0150220 2026-02-22T21:42:34.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.667+0000 7f78d6ffd640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 0x7f78c007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:34.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.668+0000 7f78f5b61640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 0x7f78c007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:34.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.668+0000 7f78f5b61640 10 monclient: get_auth_request con 0x7f78c0078e50 auth_method 0 2026-02-22T21:42:34.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.668+0000 7f78f5b61640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 0x7f78c007b310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f78e4039000 tx=0x7f78e4033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:34.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.668+0000 7f78d6ffd640 10 client.5130.objecter ms_handle_connect 0x7f78c0078e50 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.768+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f78e005d400 con 0x7f78f0150220 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.768+0000 7f78d6ffd640 10 client.5130.objecter ms_dispatch2 0x7f78f0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.768+0000 7f78d6ffd640 3 client.5130.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.768+0000 7f78d6ffd640 3 client.5130.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.769+0000 7f78d6ffd640 20 client.5130.objecter dump_active .. 0 homeless 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.769+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f78e0021020 con 0x7f78f0150220 2026-02-22T21:42:34.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.769+0000 7f78d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:34.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.769+0000 7f78d6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:34.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.869+0000 7f78f85ed640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:34.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.869+0000 7f78f85ed640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:34.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.869+0000 7f78f85ed640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:34.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.869+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f78f0074da0 con 0x7f78f0150220 2026-02-22T21:42:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.872+0000 7f78d6ffd640 1 -- 10.20.193.51:0/166612082 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f78e00613c0 con 0x7f78f0150220 2026-02-22T21:42:34.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.872+0000 7f78d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:34.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.872+0000 7f78d6ffd640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:34.874 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:34.874 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 10 librados: watch_flush enter 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 10 librados: watch_flush exit 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 20 client.5130.objecter shutdown clearing up homeless session... 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 10 client.5130.objecter successfully canceled tick 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 msgr2=0x7f78c007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 0x7f78c007b310 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f78e4039000 tx=0x7f78e4033000 comp rx=0 tx=0).stop 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 10 monclient: shutdown 2026-02-22T21:42:34.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 msgr2=0x7f78f006e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c700 tx=0x7f78e000cbd0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 shutdown_connections 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f78c0078e50 0x7f78c007b310 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f78f015c710 0x7f78f014cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78f0150600 0x7f78f01533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 --2- 10.20.193.51:0/166612082 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78f0150220 0x7f78f006e1c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 >> 10.20.193.51:0/166612082 conn(0x7f78f00cd7b0 msgr2=0x7f78f00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 shutdown_connections 2026-02-22T21:42:34.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 -- 10.20.193.51:0/166612082 wait complete. 2026-02-22T21:42:34.878 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:34.873+0000 7f78f85ed640 1 librados: shutdown 2026-02-22T21:42:37.884 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:37.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: get_monmap_and_config 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: build_initial_monmap 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: monmap: 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:37.952717+0000 2026-02-22T21:42:37.957 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:37.952717+0000 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:37.958 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 Processor -- start 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 -- start start 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: init 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient: _add_conns ranks=[1,0,3,2,5,4] 2026-02-22T21:42:37.959 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fc884150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc88415c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc8840703b0 0x7fc8840c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-a con 0x7fc8840703b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88415a920 con 0x7fc88415c710 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88415af60 con 0x7fc884150220 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fc88415b240 con 0x7fc8840703b0 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.952+0000 7fc88be43640 10 monclient(hunting): authenticate will time out at 1433.533008s 2026-02-22T21:42:37.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc88a3b9640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fc8840703b0 0x7fc8840c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc889bb8640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fc884150220 0x7fc88415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:36304/0 (socket says 10.20.193.51:36304) 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc889bb8640 1 -- 10.20.193.51:0/3096826662 learned_addr learned my addr 10.20.193.51:0/3096826662 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc88a3b9640 10 monclient(hunting): get_auth_request con 0x7fc8840703b0 auth_method 0 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc88a3b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc88a3b9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.953+0000 7fc88a3b9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 1 -- 10.20.193.51:0/3096826662 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793074609 0 0) 0x7fc88415af60 con 0x7fc884150220 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 10 monclient(hunting): _negotiate 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:37.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:37.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 10 monclient(hunting): my global_id is 5122 2026-02-22T21:42:37.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.954+0000 7fc888bb6640 1 -- 10.20.193.51:0/3096826662 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc86c003620 con 0x7fc884150220 2026-02-22T21:42:37.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_done global_id 5133 payload 274 2026-02-22T21:42:37.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3096826662 >> v1:10.20.193.68:6789/0 conn(0x7fc884150220 legacy=0x7fc88415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:37.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3096826662 >> v1:10.20.193.51:6789/0 conn(0x7fc88415c710 legacy=0x7fc88406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 1 monclient: found mon.noname-a 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.956+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3096826662 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88415b520 con 0x7fc8840703b0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.957+0000 7fc88a3b9640 1 --2- 10.20.193.51:0/3096826662 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8840703b0 0x7fc8840c73a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc880009dc0 tx=0x7fc88002f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b924626d24b113 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 1 -- 10.20.193.51:0/3096826662 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88003c070 con 0x7fc8840703b0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: dump: 2026-02-22T21:42:37.970 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: _finish_auth 0 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: _check_auth_tickets 2026-02-22T21:42:37.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 1 -- 10.20.193.51:0/3096826662 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc880037440 con 0x7fc8840703b0 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 5 monclient: authenticate success, global_id 5133 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 1 -- 10.20.193.51:0/3096826662 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880037810 con 0x7fc8840703b0 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc888bb6640 10 monclient: dump: 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.972 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 4 set_mon_vals no callback set 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 10 monclient: shutdown 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 -- 10.20.193.51:0/3096826662 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8840703b0 msgr2=0x7fc8840c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 --2- 10.20.193.51:0/3096826662 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8840703b0 0x7fc8840c73a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc880009dc0 tx=0x7fc88002f2f0 comp rx=0 tx=0).stop 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 -- 10.20.193.51:0/3096826662 shutdown_connections 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 --2- 10.20.193.51:0/3096826662 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc8840703b0 0x7fc8840c73a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:37.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 -- 10.20.193.51:0/3096826662 >> 10.20.193.51:0/3096826662 conn(0x7fc8840cd7b0 msgr2=0x7fc8840cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 -- 10.20.193.51:0/3096826662 shutdown_connections 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.959+0000 7fc88be43640 1 -- 10.20.193.51:0/3096826662 wait complete. 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 10 monclient: build_initial_monmap 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 10 monclient: monmap: 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:37.974 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 librados: starting msgr at 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 librados: starting objecter 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 Processor -- start 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 -- start start 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 librados: setting wanted keys 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 1 librados: calling monclient init 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.960+0000 7fc88be43640 10 monclient: init 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8840703b0 0x7fc88414ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-b con 0x7fc8840703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc884150220 0x7fc8841f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-c con 0x7fc884150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:37.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): picked mon.noname-a con 0x7fc88415c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fc884150600 con 0x7fc88415c710 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fc88415bae0 con 0x7fc8840703b0 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fc884150780 con 0x7fc884150220 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88be43640 10 monclient(hunting): authenticate will time out at 1433.541059s 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39462/0 (socket says 10.20.193.51:39462) 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8840703b0 0x7fc88414ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:37.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3430927079 learned_addr learned my addr 10.20.193.51:0/3430927079 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): get_auth_request con 0x7fc88415c710 auth_method 0 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc8893b7640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc884150220 0x7fc8841f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): get_auth_request con 0x7fc8840703b0 auth_method 0 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc8893b7640 10 monclient(hunting): get_auth_request con 0x7fc884150220 auth_method 0 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc8893b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:37.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc8893b7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc8893b7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.961+0000 7fc889bb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc8893b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc8893b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc8893b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 10 monclient(hunting): handle_auth_done global_id 5139 payload 824 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3430927079 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc884150220 msgr2=0x7fc8841f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc884150220 0x7fc8841f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3430927079 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8840703b0 msgr2=0x7fc88414ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8840703b0 0x7fc88414ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 monclient: found mon.noname-a 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:37.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.962+0000 7fc88a3b9640 1 -- 10.20.193.51:0/3430927079 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8841f24d0 con 0x7fc88415c710 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.963+0000 7fc88a3b9640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc88003a040 tx=0x7fc880004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.963+0000 7fc8727fc640 10 client.?.objecter ms_handle_connect 0x7fc88415c710 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.963+0000 7fc8727fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88003c050 con 0x7fc88415c710 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: dump: 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:37.979 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc880048070 con 0x7fc88415c710 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880041a30 con 0x7fc88415c710 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 5 monclient: authenticate success, global_id 5139 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc8727fc640 10 monclient: dump: 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:37.980 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc873fff640 4 set_mon_vals no callback set 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 10 monclient: _renew_subs 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8841f27b0 con 0x7fc88415c710 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 10 client.5139.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 10 monclient: _renew_subs 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc884154510 con 0x7fc88415c710 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.965+0000 7fc88be43640 1 librados: init done 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.966+0000 7fc88be43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.966+0000 7fc88be43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:37.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.966+0000 7fc88be43640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:37.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:37.966+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8840c73a0 con 0x7fc88415c710 2026-02-22T21:42:38.014 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.013+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fc880041bd0 con 0x7fc88415c710 2026-02-22T21:42:38.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.013+0000 7fc8727fc640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 0x7fc85407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:38.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.013+0000 7fc889bb8640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 0x7fc85407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:38.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.013+0000 7fc889bb8640 10 monclient: get_auth_request con 0x7fc854078e50 auth_method 0 2026-02-22T21:42:38.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.014+0000 7fc889bb8640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 0x7fc85407b310 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc874007b60 tx=0x7fc874007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:38.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.014+0000 7fc8727fc640 10 client.5139.objecter ms_handle_connect 0x7fc854078e50 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.082+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fc880084410 con 0x7fc88415c710 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.082+0000 7fc8727fc640 10 client.5139.objecter ms_dispatch2 0x7fc884000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.082+0000 7fc8727fc640 3 client.5139.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.082+0000 7fc8727fc640 3 client.5139.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.083+0000 7fc8727fc640 20 client.5139.objecter dump_active .. 0 homeless 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.083+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fc8800883d0 con 0x7fc88415c710 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.083+0000 7fc8727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:38.084 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.083+0000 7fc8727fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:38.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.182+0000 7fc88be43640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:38.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.182+0000 7fc88be43640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:38.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.182+0000 7fc88be43640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:38.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.182+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc884074da0 con 0x7fc88415c710 2026-02-22T21:42:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.184+0000 7fc8727fc640 1 -- 10.20.193.51:0/3430927079 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7fc88008c6d0 con 0x7fc88415c710 2026-02-22T21:42:38.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.184+0000 7fc8727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:38.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.184+0000 7fc8727fc640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:38.186 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:38.186 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 10 librados: watch_flush enter 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 10 librados: watch_flush exit 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 20 client.5139.objecter shutdown clearing up homeless session... 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 10 client.5139.objecter successfully canceled tick 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 msgr2=0x7fc85407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 0x7fc85407b310 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc874007b60 tx=0x7fc874007ab0 comp rx=0 tx=0).stop 2026-02-22T21:42:38.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 10 monclient: shutdown 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 msgr2=0x7fc884156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc88003a040 tx=0x7fc880004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 shutdown_connections 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fc854078e50 0x7fc85407b310 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fc88415c710 0x7fc884156b20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fc884150220 0x7fc8841f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 --2- 10.20.193.51:0/3430927079 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fc8840703b0 0x7fc88414ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 >> 10.20.193.51:0/3430927079 conn(0x7fc8840cd7b0 msgr2=0x7fc88414f540 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:38.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 shutdown_connections 2026-02-22T21:42:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 -- 10.20.193.51:0/3430927079 wait complete. 2026-02-22T21:42:38.190 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:38.185+0000 7fc88be43640 1 librados: shutdown 2026-02-22T21:42:41.196 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.264+0000 7f486372c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.264+0000 7f486372c640 10 monclient: build_initial_monmap 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.264+0000 7f486372c640 10 monclient: monmap: 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:41.265191+0000 2026-02-22T21:42:41.266 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:41.265191+0000 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:41.267 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 Processor -- start 2026-02-22T21:42:41.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 -- start start 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient: init 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient: _add_conns ranks=[1,3,4,5,0,2] 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f485c152790 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f485c152b70 0x7f485c0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): picked mon.noname-a con 0x7f485c152b70 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f485c0cc2f0 0x7f485c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): picked mon.noname-b con 0x7f485c0cc2f0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f485c0cc8b0 con 0x7f485c152790 2026-02-22T21:42:41.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f485c0c9480 con 0x7f485c152b70 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f485c0cd7a0 con 0x7f485c0cc2f0 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.265+0000 7f486372c640 10 monclient(hunting): authenticate will time out at 1436.845777s 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f485c152b70 0x7f485c0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4861ca2640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f485c0cc2f0 0x7f485c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f485c152b70 0x7f485c0ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39466/0 (socket says 10.20.193.51:39466) 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 1 -- 10.20.193.51:0/1168462393 learned_addr learned my addr 10.20.193.51:0/1168462393 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): get_auth_request con 0x7f485c152b70 auth_method 0 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:41.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4861ca2640 10 monclient(hunting): get_auth_request con 0x7f485c0cc2f0 auth_method 0 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4861ca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4861ca2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4861ca2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.266+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 10 monclient(hunting): handle_auth_done global_id 5142 payload 274 2026-02-22T21:42:41.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 1 -- 10.20.193.51:0/1168462393 >> v2:10.20.193.68:3300/0 conn(0x7f485c0cc2f0 msgr2=0x7f485c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 1 --2- 10.20.193.51:0/1168462393 >> v2:10.20.193.68:3300/0 conn(0x7f485c0cc2f0 0x7f485c0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 1 -- 10.20.193.51:0/1168462393 >> v1:10.20.193.68:6789/0 conn(0x7f485c152790 legacy=0x7f485c0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 10 monclient: _finish_hunting 0 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 1 monclient: found mon.noname-a 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.267+0000 7f4860ca0640 1 -- 10.20.193.51:0/1168462393 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c0ccb90 con 0x7f485c152b70 2026-02-22T21:42:41.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.268+0000 7f4860ca0640 1 --2- 10.20.193.51:0/1168462393 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c0ce380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f485c0cc780 tx=0x7f485002eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287b5988d27967c7 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:41.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 1 -- 10.20.193.51:0/1168462393 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485003c070 con 0x7f485c152b70 2026-02-22T21:42:41.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: dump: 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:41.281 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: _finish_auth 0 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 1 -- 10.20.193.51:0/1168462393 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f485002fbe0 con 0x7f485c152b70 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 1 -- 10.20.193.51:0/1168462393 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850037440 con 0x7f485c152b70 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 5 monclient: authenticate success, global_id 5142 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 4 set_mon_vals no callback set 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:41.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f484bfff640 10 monclient: dump: 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 10 monclient: shutdown 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 -- 10.20.193.51:0/1168462393 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 msgr2=0x7f485c0ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 --2- 10.20.193.51:0/1168462393 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c0ce380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f485c0cc780 tx=0x7f485002eed0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 -- 10.20.193.51:0/1168462393 shutdown_connections 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 --2- 10.20.193.51:0/1168462393 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c0ce380 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 --2- 10.20.193.51:0/1168462393 >> v2:10.20.193.68:3300/0 conn(0x7f485c0cc2f0 0x7f485c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 -- 10.20.193.51:0/1168462393 >> 10.20.193.51:0/1168462393 conn(0x7f485c06d9c0 msgr2=0x7f485c06ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 -- 10.20.193.51:0/1168462393 shutdown_connections 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.270+0000 7f486372c640 1 -- 10.20.193.51:0/1168462393 wait complete. 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 10 monclient: build_initial_monmap 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 10 monclient: monmap: 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:41.284 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 librados: starting msgr at 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 librados: starting objecter 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 Processor -- start 2026-02-22T21:42:41.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 -- start start 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 librados: setting wanted keys 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 1 librados: calling monclient init 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.271+0000 7f486372c640 10 monclient: init 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f485c0cc2f0 0x7f485c15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): picked mon.noname-c con 0x7f485c0cc2f0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f485c152790 0x7f485c15d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): picked mon.noname-b con 0x7f485c152790 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): picked mon.noname-a con 0x7f485c152b70 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:41.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0cd430 con 0x7f485c152b70 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0cd150 con 0x7f485c152790 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f485c0cdc30 con 0x7f485c0cc2f0 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f486372c640 10 monclient(hunting): authenticate will time out at 1436.852189s 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39480/0 (socket says 10.20.193.51:39480) 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 1 -- 10.20.193.51:0/241968480 learned_addr learned my addr 10.20.193.51:0/241968480 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): get_auth_request con 0x7f485c152b70 auth_method 0 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:41.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4860ca0640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f485c152790 0x7f485c15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f48614a1640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f485c0cc2f0 0x7f485c15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4861ca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4860ca0640 10 monclient(hunting): get_auth_request con 0x7f485c152790 auth_method 0 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4860ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4860ca0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f4860ca0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f48614a1640 10 monclient(hunting): get_auth_request con 0x7f485c0cc2f0 auth_method 0 2026-02-22T21:42:41.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f48614a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f48614a1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.272+0000 7f48614a1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f4860ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f48614a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f48614a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.273+0000 7f48614a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 10 monclient(hunting): handle_auth_done global_id 5145 payload 824 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 -- 10.20.193.51:0/241968480 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f485c0cc2f0 msgr2=0x7f485c15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f485c0cc2f0 0x7f485c15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 -- 10.20.193.51:0/241968480 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f485c152790 msgr2=0x7f485c15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f485c152790 0x7f485c15d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 10 monclient: _finish_hunting 0 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 monclient: found mon.noname-a 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.274+0000 7f4861ca2640 1 -- 10.20.193.51:0/241968480 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c14cf40 con 0x7f485c152b70 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.275+0000 7f4861ca2640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f485800efc0 tx=0x7f485800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.275+0000 7f4849ffb640 10 client.?.objecter ms_handle_connect 0x7f485c152b70 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.275+0000 7f4849ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858018020 con 0x7f485c152b70 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:41.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: dump: 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:41.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4858014480 con 0x7f485c152b70 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858014870 con 0x7f485c152b70 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f4849ffb640 10 monclient: dump: 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:41.292 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f484b7fe640 4 set_mon_vals no callback set 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 5 monclient: authenticate success, global_id 5145 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 10 monclient: _renew_subs 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485c14d220 con 0x7f485c152b70 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 10 client.5145.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 10 monclient: _renew_subs 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.293 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f485c14d7e0 con 0x7f485c152b70 2026-02-22T21:42:41.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.276+0000 7f486372c640 1 librados: init done 2026-02-22T21:42:41.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.277+0000 7f486372c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:41.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.277+0000 7f486372c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:41.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.277+0000 7f486372c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.294 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.277+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f485c0cbfc0 con 0x7f485c152b70 2026-02-22T21:42:41.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.288+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4858014a10 con 0x7f485c152b70 2026-02-22T21:42:41.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.288+0000 7f4849ffb640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 0x7f482c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:41.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.288+0000 7f48614a1640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 0x7f482c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:41.295 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.288+0000 7f48614a1640 10 monclient: get_auth_request con 0x7f482c078e00 auth_method 0 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.289+0000 7f48614a1640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 0x7f482c07b2c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f484c002770 tx=0x7f484c007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.289+0000 7f4849ffb640 10 client.5145.objecter ms_handle_connect 0x7f482c078e00 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.290+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f485805edf0 con 0x7f485c152b70 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.290+0000 7f4849ffb640 10 client.5145.objecter ms_dispatch2 0x7f485c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.290+0000 7f4849ffb640 3 client.5145.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.290+0000 7f4849ffb640 3 client.5145.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:41.296 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.290+0000 7f4849ffb640 20 client.5145.objecter dump_active .. 0 homeless 2026-02-22T21:42:41.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.391+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f485c0cbfc0 con 0x7f485c152b70 2026-02-22T21:42:41.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.391+0000 7f4849ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:41.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.391+0000 7f4849ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.490+0000 7f486372c640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.490+0000 7f486372c640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.490+0000 7f486372c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:41.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.490+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f485c0c8d70 con 0x7f485c152b70 2026-02-22T21:42:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.492+0000 7f4849ffb640 1 -- 10.20.193.51:0/241968480 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f4858022070 con 0x7f485c152b70 2026-02-22T21:42:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.492+0000 7f4849ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:41.494 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.492+0000 7f4849ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:41.494 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:41.494 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 10 librados: watch_flush enter 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 10 librados: watch_flush exit 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 20 client.5145.objecter shutdown clearing up homeless session... 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 10 client.5145.objecter successfully canceled tick 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 msgr2=0x7f482c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 0x7f482c07b2c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f484c002770 tx=0x7f484c007ef0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 10 monclient: shutdown 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 msgr2=0x7f485c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:41.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f485800efc0 tx=0x7f485800c490 comp rx=0 tx=0).stop 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 shutdown_connections 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f482c078e00 0x7f482c07b2c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f485c152b70 0x7f485c14dfe0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f485c152790 0x7f485c15d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 --2- 10.20.193.51:0/241968480 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f485c0cc2f0 0x7f485c15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 >> 10.20.193.51:0/241968480 conn(0x7f485c06d9c0 msgr2=0x7f485c1479c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 shutdown_connections 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 -- 10.20.193.51:0/241968480 wait complete. 2026-02-22T21:42:41.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:41.494+0000 7f486372c640 1 librados: shutdown 2026-02-22T21:42:41.728 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:12.881 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.754 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:12.908 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.761 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:12.915 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.781 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:12.932 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.781 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:12.897 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.782 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:12.934 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.786 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:12.902 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.790 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:12.941 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:41.794 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:12.910 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:42:44.504 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T21:42:44.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.571+0000 7f7827443640 10 monclient: get_monmap_and_config 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.571+0000 7f7827443640 10 monclient: build_initial_monmap 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.571+0000 7f7827443640 10 monclient: monmap: 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:44.572302+0000 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:44.572302+0000 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:44.573 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:44.574 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:44.574 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:44.574 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:44.574 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:44.574 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 Processor -- start 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 -- start start 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient: init 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient: _add_conns ranks=[2,1,5,3,4,0] 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7820152ab0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f782006d9d0 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f78200c95a0 0x7f78201632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): picked mon.noname-c con 0x7f78200c95a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f782006f1d0 con 0x7f782006d9d0 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f782006de50 con 0x7f7820152ab0 2026-02-22T21:42:44.577 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f782006ed40 con 0x7f78200c95a0 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.572+0000 7f7827443640 10 monclient(hunting): authenticate will time out at 1440.152663s 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f78200c95a0 0x7f78201632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78249b7640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f782006d9d0 0x7f78200c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:36328/0 (socket says 10.20.193.51:36328) 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78249b7640 1 -- 10.20.193.51:0/2554577229 learned_addr learned my addr 10.20.193.51:0/2554577229 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78251b8640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f7820152ab0 0x7f78200c5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:46010/0 (socket says 10.20.193.51:46010) 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): get_auth_request con 0x7f78200c95a0 auth_method 0 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.573+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 1 -- 10.20.193.51:0/2554577229 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653066567 0 0) 0x7f782006f1d0 con 0x7f782006d9d0 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 10 monclient(hunting): _negotiate 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 10 monclient(hunting): my global_id is 5134 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.574+0000 7f780ffff640 1 -- 10.20.193.51:0/2554577229 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7808003620 con 0x7f782006d9d0 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 10 monclient(hunting): handle_auth_done global_id 5105 payload 274 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 1 -- 10.20.193.51:0/2554577229 >> v1:10.20.193.125:6789/0 conn(0x7f7820152ab0 legacy=0x7f78200c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 1 -- 10.20.193.51:0/2554577229 >> v1:10.20.193.68:6789/0 conn(0x7f782006d9d0 legacy=0x7f78200c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 1 monclient: found mon.noname-c 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.575+0000 7f78259b9640 1 -- 10.20.193.51:0/2554577229 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782006e130 con 0x7f78200c95a0 2026-02-22T21:42:44.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.577+0000 7f78259b9640 1 --2- 10.20.193.51:0/2554577229 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78200c95a0 0x7f78201632e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f781c009dc0 tx=0x7f781c02f2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dbf7de216c6b1bc server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 1 -- 10.20.193.51:0/2554577229 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c03c070 con 0x7f78200c95a0 2026-02-22T21:42:44.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:44.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: dump: 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: _finish_auth 0 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 1 -- 10.20.193.51:0/2554577229 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781c037440 con 0x7f78200c95a0 2026-02-22T21:42:44.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 5 monclient: authenticate success, global_id 5105 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 1 -- 10.20.193.51:0/2554577229 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c037810 con 0x7f78200c95a0 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 4 set_mon_vals no callback set 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f780ffff640 10 monclient: dump: 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.583 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 10 monclient: shutdown 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 -- 10.20.193.51:0/2554577229 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78200c95a0 msgr2=0x7f78201632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 --2- 10.20.193.51:0/2554577229 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78200c95a0 0x7f78201632e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f781c009dc0 tx=0x7f781c02f2f0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 -- 10.20.193.51:0/2554577229 shutdown_connections 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 --2- 10.20.193.51:0/2554577229 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f78200c95a0 0x7f78201632e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 -- 10.20.193.51:0/2554577229 >> 10.20.193.51:0/2554577229 conn(0x7f78200cd7b0 msgr2=0x7f78200cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 -- 10.20.193.51:0/2554577229 shutdown_connections 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.579+0000 7f7827443640 1 -- 10.20.193.51:0/2554577229 wait complete. 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 10 monclient: build_initial_monmap 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 10 monclient: monmap: 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:44.584 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.585 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:44.585 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:44.585 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:44.585 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 librados: starting msgr at 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 librados: starting objecter 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 Processor -- start 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 -- start start 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 librados: setting wanted keys 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 1 librados: calling monclient init 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.580+0000 7f7827443640 10 monclient: init 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f782006d9d0 0x7f78200cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): picked mon.noname-b con 0x7f782006d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78200c95a0 0x7f78201f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): picked mon.noname-a con 0x7f78200c95a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): picked mon.noname-c con 0x7f7820152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f782006e9d0 con 0x7f78200c95a0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f782006e6f0 con 0x7f782006d9d0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f78200c61c0 con 0x7f7820152ab0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f7827443640 10 monclient(hunting): authenticate will time out at 1440.161148s 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78251b8640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f782006d9d0 0x7f78200cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78249b7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78200c95a0 0x7f78201f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:40412/0 (socket says 10.20.193.51:40412) 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 1 -- 10.20.193.51:0/1935748044 learned_addr learned my addr 10.20.193.51:0/1935748044 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): get_auth_request con 0x7f7820152ab0 auth_method 0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78249b7640 10 monclient(hunting): get_auth_request con 0x7f78200c95a0 auth_method 0 2026-02-22T21:42:44.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78249b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78249b7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78249b7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78251b8640 10 monclient(hunting): get_auth_request con 0x7f782006d9d0 auth_method 0 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78251b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78251b8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78251b8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.581+0000 7f78259b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78249b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78249b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78249b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78251b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78251b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.582+0000 7f78251b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 10 monclient(hunting): handle_auth_done global_id 5111 payload 824 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 -- 10.20.193.51:0/1935748044 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f782006d9d0 msgr2=0x7f78200cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f782006d9d0 0x7f78200cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 -- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78200c95a0 msgr2=0x7f78201f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78200c95a0 0x7f78201f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 monclient: found mon.noname-c 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.583+0000 7f78259b9640 1 -- 10.20.193.51:0/1935748044 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78200c3ba0 con 0x7f7820152ab0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.584+0000 7f78259b9640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f781c03a040 tx=0x7f781c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.584+0000 7f780dffb640 10 client.?.objecter ms_handle_connect 0x7f7820152ab0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.584+0000 7f780dffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c03c050 con 0x7f7820152ab0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: dump: 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.591 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781c048070 con 0x7f7820152ab0 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c041a30 con 0x7f7820152ab0 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780dffb640 10 monclient: dump: 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.592 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f780f7fe640 4 set_mon_vals no callback set 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 5 monclient: authenticate success, global_id 5111 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 10 monclient: _renew_subs 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78200c3e80 con 0x7f7820152ab0 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 10 client.5111.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 10 monclient: _renew_subs 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78201f3300 con 0x7f7820152ab0 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.585+0000 7f7827443640 1 librados: init done 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.586+0000 7f7827443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.586+0000 7f7827443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.586+0000 7f7827443640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.593 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.586+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78200c41e0 con 0x7f7820152ab0 2026-02-22T21:42:44.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.599+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f781c041bd0 con 0x7f7820152ab0 2026-02-22T21:42:44.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.600+0000 7f780dffb640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 0x7f77f007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.600+0000 7f78251b8640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 0x7f77f007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.600+0000 7f78251b8640 10 monclient: get_auth_request con 0x7f77f0078e50 auth_method 0 2026-02-22T21:42:44.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.600+0000 7f780dffb640 10 client.5111.objecter ms_handle_connect 0x7f77f0078e50 2026-02-22T21:42:44.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.600+0000 7f78251b8640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 0x7f77f007b310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7810002770 tx=0x7f7810007f20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.602+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f781c084400 con 0x7f7820152ab0 2026-02-22T21:42:44.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.602+0000 7f780dffb640 10 client.5111.objecter ms_dispatch2 0x7f7820000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:44.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.602+0000 7f780dffb640 3 client.5111.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:44.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.602+0000 7f780dffb640 3 client.5111.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:44.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.602+0000 7f780dffb640 20 client.5111.objecter dump_active .. 0 homeless 2026-02-22T21:42:44.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.701+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f78200c41e0 con 0x7f7820152ab0 2026-02-22T21:42:44.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.701+0000 7f780dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:44.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.701+0000 7f780dffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:44.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.802+0000 7f7827443640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2026-02-22T21:42:44.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.802+0000 7f7827443640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:44.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.802+0000 7f7827443640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:44.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.802+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f782006d790 con 0x7f7820152ab0 2026-02-22T21:42:44.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.806+0000 7f780dffb640 1 -- 10.20.193.51:0/1935748044 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+201 (secure 0 0 0) 0x7f782006d790 con 0x7f7820152ab0 2026-02-22T21:42:44.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.806+0000 7f780dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2026-02-22T21:42:44.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.806+0000 7f780dffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:44.808 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:44.808 INFO:teuthology.orchestra.run.trial051.stdout:{"status":"HEALTH_WARN","checks":{"BLUESTORE_SLOW_OP_ALERT":{"severity":"HEALTH_WARN","summary":{"message":"12 OSD(s) experiencing slow operations in BlueStore","count":12},"muted":false}},"mutes":[]} 2026-02-22T21:42:44.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 10 librados: watch_flush enter 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 10 librados: watch_flush exit 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 20 client.5111.objecter shutdown clearing up homeless session... 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 10 client.5111.objecter successfully canceled tick 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 msgr2=0x7f77f007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 0x7f77f007b310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7810002770 tx=0x7f7810007f20 comp rx=0 tx=0).stop 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 10 monclient: shutdown 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 msgr2=0x7f78201f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f781c03a040 tx=0x7f781c004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 shutdown_connections 2026-02-22T21:42:44.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f77f0078e50 0x7f77f007b310 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7820152ab0 0x7f78201f4600 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f78200c95a0 0x7f78201f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 --2- 10.20.193.51:0/1935748044 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f782006d9d0 0x7f78200cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 >> 10.20.193.51:0/1935748044 conn(0x7f78200cd7b0 msgr2=0x7f78200cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 shutdown_connections 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 -- 10.20.193.51:0/1935748044 wait complete. 2026-02-22T21:42:44.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.807+0000 7f7827443640 1 librados: shutdown 2026-02-22T21:42:44.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T21:42:44.817 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.884+0000 7efea5133640 10 monclient: get_monmap_and_config 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.884+0000 7efea5133640 10 monclient: build_initial_monmap 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.884+0000 7efea5133640 10 monclient: monmap: 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:44.885647+0000 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:44.885647+0000 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:44.886 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.885+0000 7efea5133640 1 Processor -- start 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.885+0000 7efea5133640 1 -- start start 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.885+0000 7efea5133640 10 monclient: init 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient: _add_conns ranks=[3,5,1,2,0,4] 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7efea0150220 0x7efea00c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): picked mon.noname-a con 0x7efea0150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7efea0150600 0x7efea015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): picked mon.noname-c con 0x7efea0150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7efea015c540 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea015c9b0 con 0x7efea015c540 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7efea015a920 con 0x7efea0150220 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7efea015af60 con 0x7efea0150600 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efea5133640 10 monclient(hunting): authenticate will time out at 1440.466072s 2026-02-22T21:42:44.887 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7efea0150220 0x7efea00c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7efea0150220 0x7efea00c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39496/0 (socket says 10.20.193.51:39496) 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 1 -- 10.20.193.51:0/716524231 learned_addr learned my addr 10.20.193.51:0/716524231 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): get_auth_request con 0x7efea0150220 auth_method 0 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9e575640 1 --2- 10.20.193.51:0/716524231 >> v2:10.20.193.125:3300/0 conn(0x7efea0150600 0x7efea015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9e575640 10 monclient(hunting): get_auth_request con 0x7efea0150600 auth_method 0 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.886+0000 7efe9e575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 10 monclient(hunting): handle_auth_done global_id 5151 payload 274 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 1 -- 10.20.193.51:0/716524231 >> v2:10.20.193.125:3300/0 conn(0x7efea0150600 msgr2=0x7efea015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 1 --2- 10.20.193.51:0/716524231 >> v2:10.20.193.125:3300/0 conn(0x7efea0150600 0x7efea015c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 1 -- 10.20.193.51:0/716524231 >> v1:10.20.193.68:6789/0 conn(0x7efea015c540 legacy=0x7efea006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 10 monclient: _finish_hunting 0 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 1 monclient: found mon.noname-a 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.887+0000 7efe9ed76640 1 -- 10.20.193.51:0/716524231 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea015b240 con 0x7efea0150220 2026-02-22T21:42:44.889 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.888+0000 7efe9ed76640 1 --2- 10.20.193.51:0/716524231 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150220 0x7efea00c73a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efea0074940 tx=0x7efe9402ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96da75bae5671f56 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 1 -- 10.20.193.51:0/716524231 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9403c070 con 0x7efea0150220 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 10 monclient: dump: 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.890 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 10 monclient: _finish_auth 0 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 10 monclient: _check_auth_tickets 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efe9dd74640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efea5133640 5 monclient: authenticate success, global_id 5151 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.889+0000 7efea5133640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 1 -- 10.20.193.51:0/716524231 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9402fc70 con 0x7efea0150220 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 1 -- 10.20.193.51:0/716524231 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe94037660 con 0x7efea0150220 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efe9dd74640 10 monclient: dump: 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.891 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 4 set_mon_vals no callback set 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 10 monclient: shutdown 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 -- 10.20.193.51:0/716524231 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150220 msgr2=0x7efea00c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 --2- 10.20.193.51:0/716524231 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150220 0x7efea00c73a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efea0074940 tx=0x7efe9402ef60 comp rx=0 tx=0).stop 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 -- 10.20.193.51:0/716524231 shutdown_connections 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 --2- 10.20.193.51:0/716524231 >> v2:10.20.193.125:3300/0 conn(0x7efea0150600 0x7efea015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 --2- 10.20.193.51:0/716524231 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150220 0x7efea00c73a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 -- 10.20.193.51:0/716524231 >> 10.20.193.51:0/716524231 conn(0x7efea00cd7b0 msgr2=0x7efea00cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 -- 10.20.193.51:0/716524231 shutdown_connections 2026-02-22T21:42:44.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 1 -- 10.20.193.51:0/716524231 wait complete. 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 10 monclient: build_initial_monmap 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.890+0000 7efea5133640 10 monclient: monmap: 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 librados: starting msgr at 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 librados: starting objecter 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 Processor -- start 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 -- start start 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 librados: setting wanted keys 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 librados: calling monclient init 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient: init 2026-02-22T21:42:44.893 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efea0150220 0x7efea01f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): picked mon.noname-b con 0x7efea0150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): picked mon.noname-a con 0x7efea0150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efea015c540 0x7efea0073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): picked mon.noname-c con 0x7efea015c540 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7efea015bae0 con 0x7efea0150600 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7efea015b800 con 0x7efea0150220 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7efea00703b0 con 0x7efea015c540 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efea5133640 10 monclient(hunting): authenticate will time out at 1440.471538s 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39502/0 (socket says 10.20.193.51:39502) 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 1 -- 10.20.193.51:0/1438165718 learned_addr learned my addr 10.20.193.51:0/1438165718 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 10 monclient(hunting): get_auth_request con 0x7efea0150600 auth_method 0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9e575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9f577640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efea015c540 0x7efea0073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.891+0000 7efe9ed76640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efea0150220 0x7efea01f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9e575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): get_auth_request con 0x7efea015c540 auth_method 0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): get_auth_request con 0x7efea0150220 auth_method 0 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:44.894 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9f577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.892+0000 7efe9ed76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 10 monclient(hunting): handle_auth_done global_id 5154 payload 824 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 -- 10.20.193.51:0/1438165718 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efea015c540 msgr2=0x7efea0073220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efea015c540 0x7efea0073220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 -- 10.20.193.51:0/1438165718 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efea0150220 msgr2=0x7efea01f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efea0150220 0x7efea01f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 10 monclient: _finish_hunting 0 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 monclient: found mon.noname-a 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.893+0000 7efe9e575640 1 -- 10.20.193.51:0/1438165718 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea00724a0 con 0x7efea0150600 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.894+0000 7efe9e575640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe8800ebe0 tx=0x7efe8800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.894+0000 7efe7f7fe640 10 client.?.objecter ms_handle_connect 0x7efea0150600 2026-02-22T21:42:44.895 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.894+0000 7efe7f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88017070 con 0x7efea0150600 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: dump: 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:44.897 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8800cc40 con 0x7efea0150600 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88013470 con 0x7efea0150600 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe7f7fe640 10 monclient: dump: 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efe9d573640 4 set_mon_vals no callback set 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 5 monclient: authenticate success, global_id 5154 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _renew_subs 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea014cf60 con 0x7efea0150600 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 client.5154.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _renew_subs 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea0072a20 con 0x7efea0150600 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 1 librados: init done 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:44.898 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:44.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:44.899 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.896+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea015c000 con 0x7efea0150600 2026-02-22T21:42:44.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.908+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7efe88024060 con 0x7efea0150600 2026-02-22T21:42:44.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.908+0000 7efe7f7fe640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 0x7efe6c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:44.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.908+0000 7efe9ed76640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 0x7efe6c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:44.909 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.908+0000 7efe9ed76640 10 monclient: get_auth_request con 0x7efe6c078e50 auth_method 0 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe9ed76640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 0x7efe6c07b310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efea00c65c0 tx=0x7efe94039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 10 client.5154.objecter ms_handle_connect 0x7efe6c078e50 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7efe8805e170 con 0x7efea0150600 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 10 client.5154.objecter ms_dispatch2 0x7efea0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 3 client.5154.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 3 client.5154.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:44.910 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:44.909+0000 7efe7f7fe640 20 client.5154.objecter dump_active .. 0 homeless 2026-02-22T21:42:45.010 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.009+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7efe88062130 con 0x7efea0150600 2026-02-22T21:42:45.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.009+0000 7efe7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.009+0000 7efe7f7fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:45.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.090+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efea0070dd0 con 0x7efe6c078e50 2026-02-22T21:42:45.092 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.091+0000 7efe7f7fe640 1 -- 10.20.193.51:0/1438165718 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44291 (secure 0 0 0) 0x7efea0070dd0 con 0x7efe6c078e50 2026-02-22T21:42:45.093 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 10 librados: watch_flush enter 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 10 librados: watch_flush exit 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 20 client.5154.objecter shutdown clearing up homeless session... 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 10 client.5154.objecter successfully canceled tick 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 msgr2=0x7efe6c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 0x7efe6c07b310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efea00c65c0 tx=0x7efe94039040 comp rx=0 tx=0).stop 2026-02-22T21:42:45.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 10 monclient: shutdown 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 msgr2=0x7efea00ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efe8800ebe0 tx=0x7efe8800c520 comp rx=0 tx=0).stop 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 shutdown_connections 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7efe6c078e50 0x7efe6c07b310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7efea015c540 0x7efea0073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7efea0150600 0x7efea00ca2a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 --2- 10.20.193.51:0/1438165718 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7efea0150220 0x7efea01f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 >> 10.20.193.51:0/1438165718 conn(0x7efea00cd7b0 msgr2=0x7efea00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 shutdown_connections 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 -- 10.20.193.51:0/1438165718 wait complete. 2026-02-22T21:42:45.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.092+0000 7efea5133640 1 librados: shutdown 2026-02-22T21:42:45.102 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-22T21:42:43.765285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437528,"kb_used_data":6104,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509511912,"statfs":{"total":1546188226560,"available":1545740197888,"internally_reserved":0,"allocated":6250496,"data_stored":3091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001911"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:44.312096+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:39:44.312096+0000","last_peered":"2026-02-22T21:39:44.312096+0000","last_clean":"2026-02-22T21:39:44.312096+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:39:44.312096+0000","last_undegraded":"2026-02-22T21:39:44.312096+0000","last_fullsized":"2026-02-22T21:39:44.312096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:42.253559+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:39:42.253559+0000","last_peered":"2026-02-22T21:39:42.253559+0000","last_clean":"2026-02-22T21:39:42.253559+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:39:42.253559+0000","last_undegraded":"2026-02-22T21:39:42.253559+0000","last_fullsized":"2026-02-22T21:39:42.253559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:40.400160+0000","last_change":"2026-02-22T21:37:38.380932+0000","last_active":"2026-02-22T21:39:40.400160+0000","last_peered":"2026-02-22T21:39:40.400160+0000","last_clean":"2026-02-22T21:39:40.400160+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:39:40.400160+0000","last_undegraded":"2026-02-22T21:39:40.400160+0000","last_fullsized":"2026-02-22T21:39:40.400160+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:22:09.624552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:37.344398+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:39:37.344398+0000","last_peered":"2026-02-22T21:39:37.344398+0000","last_clean":"2026-02-22T21:39:37.344398+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:39:37.344398+0000","last_undegraded":"2026-02-22T21:39:37.344398+0000","last_fullsized":"2026-02-22T21:39:37.344398+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"30'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:39.310121+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:39:39.310121+0000","last_peered":"2026-02-22T21:39:39.310121+0000","last_clean":"2026-02-22T21:39:39.310121+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:39:39.310121+0000","last_undegraded":"2026-02-22T21:39:39.310121+0000","last_fullsized":"2026-02-22T21:39:39.310121+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437076+0000","last_change":"2026-02-22T21:37:36.829493+0000","last_active":"2026-02-22T21:39:38.437076+0000","last_peered":"2026-02-22T21:39:38.437076+0000","last_clean":"2026-02-22T21:39:38.437076+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:39:38.437076+0000","last_undegraded":"2026-02-22T21:39:38.437076+0000","last_fullsized":"2026-02-22T21:39:38.437076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:05.328006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:33.547138+0000","last_change":"2026-02-22T21:37:38.428511+0000","last_active":"2026-02-22T21:39:33.547138+0000","last_peered":"2026-02-22T21:39:33.547138+0000","last_clean":"2026-02-22T21:39:33.547138+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:39:33.547138+0000","last_undegraded":"2026-02-22T21:39:33.547138+0000","last_fullsized":"2026-02-22T21:39:33.547138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:54:42.502365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437292+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:39:38.437292+0000","last_peered":"2026-02-22T21:39:38.437292+0000","last_clean":"2026-02-22T21:39:38.437292+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:39:38.437292+0000","last_undegraded":"2026-02-22T21:39:38.437292+0000","last_fullsized":"2026-02-22T21:39:38.437292+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:04:41.120271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"30'32","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:28.703892+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:39:28.703892+0000","last_peered":"2026-02-22T21:39:28.703892+0000","last_clean":"2026-02-22T21:39:28.703892+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:39:28.703892+0000","last_undegraded":"2026-02-22T21:39:28.703892+0000","last_fullsized":"2026-02-22T21:39:28.703892+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:42:45.104 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 10 monclient: get_monmap_and_config 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 10 monclient: monmap: 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:45.170944+0000 2026-02-22T21:42:45.172 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:45.170944+0000 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 1 Processor -- start 2026-02-22T21:42:45.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 1 -- start start 2026-02-22T21:42:45.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.170+0000 7f408696a640 10 monclient: init 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient: _add_conns ranks=[1,2,4,3,0,5] 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f4080152ab0 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f408006d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f40800c95a0 0x7f40801632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): picked mon.noname-b con 0x7f40800c95a0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:45.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408006f1d0 con 0x7f4080152ab0 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408006de50 con 0x7f408006d9d0 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f408006ed40 con 0x7f40800c95a0 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f408696a640 10 monclient(hunting): authenticate will time out at 1440.751270s 2026-02-22T21:42:45.180 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f407f7fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f408006d9d0 0x7f40800c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:46018/0 (socket says 10.20.193.51:46018) 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.171+0000 7f407f7fe640 1 -- 10.20.193.51:0/521495352 learned_addr learned my addr 10.20.193.51:0/521495352 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.172+0000 7f4084ee0640 1 --2- 10.20.193.51:0/521495352 >> v2:10.20.193.68:3300/0 conn(0x7f40800c95a0 0x7f40801632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745425315 0 0) 0x7f408006de50 con 0x7f408006d9d0 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): my global_id is 5120 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4064003620 con 0x7f408006d9d0 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): get_auth_request con 0x7f40800c95a0 auth_method 0 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.181 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700382773 0 0) 0x7f408006f1d0 con 0x7f4080152ab0 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 10 monclient(hunting): my global_id is 5146 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f408006de50 con 0x7f4080152ab0 2026-02-22T21:42:45.182 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.173+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2435356139 0 0) 0x7f4064003620 con 0x7f408006d9d0 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 >> v2:10.20.193.68:3300/0 conn(0x7f40800c95a0 msgr2=0x7f40801632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 --2- 10.20.193.51:0/521495352 >> v2:10.20.193.68:3300/0 conn(0x7f40800c95a0 0x7f40801632e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 >> v1:10.20.193.68:6789/0 conn(0x7f4080152ab0 legacy=0x7f40800c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.183 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408006e130 con 0x7f408006d9d0 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f4070002f00 con 0x7f408006d9d0 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: dump: 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.184 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.185 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 >> v1:10.20.193.125:6789/0 conn(0x7f408006d9d0 legacy=0x7f40800c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064003d80 0x7f4064004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): picked mon.b con 0x7f4064003d80 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 0x7f4064004d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): picked mon.a con 0x7f4064004900 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4064005330 0x7f406400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): picked mon.c con 0x7f4064005330 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f408006ed40 con 0x7f4064004900 2026-02-22T21:42:45.186 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f406400c290 con 0x7f4064003d80 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f406400e720 con 0x7f4064005330 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.174+0000 7f407effd640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 0x7f4064004d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): get_auth_request con 0x7f4064004900 auth_method 0 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.187 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.175+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4064005330 0x7f406400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 10 monclient(hunting): get_auth_request con 0x7f4064005330 auth_method 0 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 10 monclient(hunting): handle_auth_done global_id 5120 payload 293 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 -- 10.20.193.51:0/521495352 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4064005330 msgr2=0x7f406400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4064005330 0x7f406400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 -- 10.20.193.51:0/521495352 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064003d80 msgr2=0x7f4064004180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064003d80 0x7f4064004180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.188 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 monclient: found mon.a 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 1 -- 10.20.193.51:0/521495352 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4064003620 con 0x7f4064004900 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f407f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f408696a640 5 monclient: authenticate success, global_id 5120 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f408696a640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.176+0000 7f4084ee0640 1 -- 10.20.193.51:0/521495352 reap_dead start 2026-02-22T21:42:45.189 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.177+0000 7f407f7fe640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 0x7f4064004d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f40800c43e0 tx=0x7f407002f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd71d7480414e253 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007da0 con 0x7f4064004900 2026-02-22T21:42:45.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.196 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: dump: 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.197 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4070005bf0 con 0x7f4064004900 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 1 -- 10.20.193.51:0/521495352 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007070 con 0x7f4064004900 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f407effd640 10 monclient: dump: 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.198 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 4 set_mon_vals no callback set 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 10 monclient: shutdown 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 1 -- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 msgr2=0x7f4064004d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.199 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 0x7f4064004d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f40800c43e0 tx=0x7f407002f560 comp rx=0 tx=0).stop 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 1 -- 10.20.193.51:0/521495352 shutdown_connections 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 1 --2- 10.20.193.51:0/521495352 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4064004900 0x7f4064004d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.179+0000 7f408696a640 1 -- 10.20.193.51:0/521495352 >> 10.20.193.51:0/521495352 conn(0x7f40800cd7b0 msgr2=0x7f40800cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 -- 10.20.193.51:0/521495352 shutdown_connections 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 -- 10.20.193.51:0/521495352 wait complete. 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 10 monclient: monmap: 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:45.200 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 librados: starting msgr at 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 librados: starting objecter 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 Processor -- start 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 -- start start 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 librados: setting wanted keys 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 1 librados: calling monclient init 2026-02-22T21:42:45.201 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.180+0000 7f408696a640 10 monclient: init 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064004900 0x7f40800bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): picked mon.noname-b con 0x7f4064004900 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): picked mon.noname-a con 0x7f40800c0010 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f40801f7c30 0x7f4080073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): picked mon.noname-c con 0x7f40801f7c30 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f408006e9d0 con 0x7f40800c0010 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.202 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f408006e6f0 con 0x7f4064004900 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f4080152e90 con 0x7f40801f7c30 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f408696a640 10 monclient(hunting): authenticate will time out at 1440.761338s 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39522/0 (socket says 10.20.193.51:39522) 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 1 -- 10.20.193.51:0/4160836190 learned_addr learned my addr 10.20.193.51:0/4160836190 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): get_auth_request con 0x7f40800c0010 auth_method 0 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f4084ee0640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f40801f7c30 0x7f4080073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.203 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407ffff640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064004900 0x7f40800bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f4084ee0640 10 monclient(hunting): get_auth_request con 0x7f40801f7c30 auth_method 0 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f4084ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f4084ee0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f4084ee0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407ffff640 10 monclient(hunting): get_auth_request con 0x7f4064004900 auth_method 0 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.181+0000 7f407ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.204 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f4084ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f407ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f407ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.182+0000 7f407ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 10 monclient(hunting): handle_auth_done global_id 5157 payload 824 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 -- 10.20.193.51:0/4160836190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f40801f7c30 msgr2=0x7f4080073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f40801f7c30 0x7f4080073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 -- 10.20.193.51:0/4160836190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064004900 msgr2=0x7f40800bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064004900 0x7f40800bfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.183+0000 7f407f7fe640 1 -- 10.20.193.51:0/4160836190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080074020 con 0x7f40800c0010 2026-02-22T21:42:45.205 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.184+0000 7f407f7fe640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f407002fa70 tx=0x7f408006e300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.184+0000 7f407cff9640 10 client.?.objecter ms_handle_connect 0x7f40800c0010 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.184+0000 7f407cff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40700033f0 con 0x7f40800c0010 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: dump: 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.206 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4070004ae0 con 0x7f40800c0010 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 5 monclient: authenticate success, global_id 5157 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 10 monclient: _renew_subs 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4080074300 con 0x7f40800c0010 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007da0 con 0x7f40800c0010 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:45.207 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407cff9640 10 monclient: dump: 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f407e7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 10 client.5157.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 10 monclient: _renew_subs 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40801f5180 con 0x7f40800c0010 2026-02-22T21:42:45.208 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.185+0000 7f408696a640 1 librados: init done 2026-02-22T21:42:45.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.186+0000 7f408696a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.186+0000 7f408696a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.186+0000 7f408696a640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:45.209 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.186+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40800c5ab0 con 0x7f40800c0010 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.196+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4070037430 con 0x7f40800c0010 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.197+0000 7f407cff9640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 0x7f405007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.197+0000 7f407ffff640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 0x7f405007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.197+0000 7f407ffff640 10 monclient: get_auth_request con 0x7f4050078e50 auth_method 0 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.197+0000 7f407ffff640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 0x7f405007b310 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f406c002770 tx=0x7f406c007e50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.197+0000 7f407cff9640 10 client.5157.objecter ms_handle_connect 0x7f4050078e50 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.198+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f4070082c70 con 0x7f40800c0010 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.198+0000 7f407cff9640 10 client.5157.objecter ms_dispatch2 0x7f4080000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:45.210 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.198+0000 7f407cff9640 3 client.5157.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:45.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.198+0000 7f407cff9640 3 client.5157.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:45.211 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.198+0000 7f407cff9640 20 client.5157.objecter dump_active .. 0 homeless 2026-02-22T21:42:45.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.296+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f4070086c30 con 0x7f40800c0010 2026-02-22T21:42:45.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.296+0000 7f407cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.296+0000 7f407cff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:45.383 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.381+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40800c6ea0 con 0x7f4050078e50 2026-02-22T21:42:45.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.382+0000 7f407cff9640 1 -- 10.20.193.51:0/4160836190 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44291 (secure 0 0 0) 0x7f40800c6ea0 con 0x7f4050078e50 2026-02-22T21:42:45.385 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:45.386 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 10 librados: watch_flush enter 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 10 librados: watch_flush exit 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 20 client.5157.objecter shutdown clearing up homeless session... 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 10 client.5157.objecter successfully canceled tick 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 msgr2=0x7f405007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 0x7f405007b310 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f406c002770 tx=0x7f406c007e50 comp rx=0 tx=0).stop 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 10 monclient: shutdown 2026-02-22T21:42:45.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 msgr2=0x7f40801f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f407002fa70 tx=0x7f408006e300 comp rx=0 tx=0).stop 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 shutdown_connections 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f4050078e50 0x7f405007b310 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f40801f7c30 0x7f4080073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f40800c0010 0x7f40801f76f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 --2- 10.20.193.51:0/4160836190 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4064004900 0x7f40800bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 >> 10.20.193.51:0/4160836190 conn(0x7f40800cd7b0 msgr2=0x7f4080151700 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 shutdown_connections 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 -- 10.20.193.51:0/4160836190 wait complete. 2026-02-22T21:42:45.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.384+0000 7f408696a640 1 librados: shutdown 2026-02-22T21:42:45.394 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-22T21:42:43.765285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437528,"kb_used_data":6104,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509511912,"statfs":{"total":1546188226560,"available":1545740197888,"internally_reserved":0,"allocated":6250496,"data_stored":3091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001911"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:44.312096+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:39:44.312096+0000","last_peered":"2026-02-22T21:39:44.312096+0000","last_clean":"2026-02-22T21:39:44.312096+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:39:44.312096+0000","last_undegraded":"2026-02-22T21:39:44.312096+0000","last_fullsized":"2026-02-22T21:39:44.312096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:42.253559+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:39:42.253559+0000","last_peered":"2026-02-22T21:39:42.253559+0000","last_clean":"2026-02-22T21:39:42.253559+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:39:42.253559+0000","last_undegraded":"2026-02-22T21:39:42.253559+0000","last_fullsized":"2026-02-22T21:39:42.253559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:40.400160+0000","last_change":"2026-02-22T21:37:38.380932+0000","last_active":"2026-02-22T21:39:40.400160+0000","last_peered":"2026-02-22T21:39:40.400160+0000","last_clean":"2026-02-22T21:39:40.400160+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:39:40.400160+0000","last_undegraded":"2026-02-22T21:39:40.400160+0000","last_fullsized":"2026-02-22T21:39:40.400160+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:22:09.624552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:37.344398+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:39:37.344398+0000","last_peered":"2026-02-22T21:39:37.344398+0000","last_clean":"2026-02-22T21:39:37.344398+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:39:37.344398+0000","last_undegraded":"2026-02-22T21:39:37.344398+0000","last_fullsized":"2026-02-22T21:39:37.344398+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"30'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:39.310121+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:39:39.310121+0000","last_peered":"2026-02-22T21:39:39.310121+0000","last_clean":"2026-02-22T21:39:39.310121+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:39:39.310121+0000","last_undegraded":"2026-02-22T21:39:39.310121+0000","last_fullsized":"2026-02-22T21:39:39.310121+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437076+0000","last_change":"2026-02-22T21:37:36.829493+0000","last_active":"2026-02-22T21:39:38.437076+0000","last_peered":"2026-02-22T21:39:38.437076+0000","last_clean":"2026-02-22T21:39:38.437076+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:39:38.437076+0000","last_undegraded":"2026-02-22T21:39:38.437076+0000","last_fullsized":"2026-02-22T21:39:38.437076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:05.328006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:33.547138+0000","last_change":"2026-02-22T21:37:38.428511+0000","last_active":"2026-02-22T21:39:33.547138+0000","last_peered":"2026-02-22T21:39:33.547138+0000","last_clean":"2026-02-22T21:39:33.547138+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:39:33.547138+0000","last_undegraded":"2026-02-22T21:39:33.547138+0000","last_fullsized":"2026-02-22T21:39:33.547138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:54:42.502365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437292+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:39:38.437292+0000","last_peered":"2026-02-22T21:39:38.437292+0000","last_clean":"2026-02-22T21:39:38.437292+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:39:38.437292+0000","last_undegraded":"2026-02-22T21:39:38.437292+0000","last_fullsized":"2026-02-22T21:39:38.437292+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:04:41.120271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"30'32","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:28.703892+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:39:28.703892+0000","last_peered":"2026-02-22T21:39:28.703892+0000","last_clean":"2026-02-22T21:39:28.703892+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:39:28.703892+0000","last_undegraded":"2026-02-22T21:39:28.703892+0000","last_fullsized":"2026-02-22T21:39:28.703892+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:42:45.396 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T21:42:45.396 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:42:45.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.463+0000 7fd4b41ed640 10 monclient: get_monmap_and_config 2026-02-22T21:42:45.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.463+0000 7fd4b41ed640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.465 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.463+0000 7fd4b41ed640 10 monclient: monmap: 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:45.464253+0000 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:45.464253+0000 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:45.466 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:45.467 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.463+0000 7fd4b41ed640 1 Processor -- start 2026-02-22T21:42:45.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 1 -- start start 2026-02-22T21:42:45.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient: init 2026-02-22T21:42:45.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient: _add_conns ranks=[0,5,2,3,4,1] 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4ac152ab0 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fd4ac06d9d0 0x7fd4ac0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ac06d9d0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4ac0c93d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4ac06f1d0 con 0x7fd4ac152ab0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4ac06de50 con 0x7fd4ac0c93d0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fd4ac06ed40 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b41ed640 10 monclient(hunting): authenticate will time out at 1441.044562s 2026-02-22T21:42:45.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b1f62640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fd4ac152ab0 0x7fd4ac0c5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:54328/0 (socket says 10.20.193.51:54328) 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.464+0000 7fd4b1f62640 1 -- 10.20.193.51:0/2603949535 learned_addr learned my addr 10.20.193.51:0/2603949535 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 1 --2- 10.20.193.51:0/2603949535 >> v2:10.20.193.125:3300/0 conn(0x7fd4ac06d9d0 0x7fd4ac0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): get_auth_request con 0x7fd4ac06d9d0 auth_method 0 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.465+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 1 -- 10.20.193.51:0/2603949535 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606043544 0 0) 0x7fd4ac06f1d0 con 0x7fd4ac152ab0 2026-02-22T21:42:45.471 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 10 monclient(hunting): _negotiate 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 10 monclient(hunting): my global_id is 5160 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b0f60640 1 -- 10.20.193.51:0/2603949535 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd494003620 con 0x7fd4ac152ab0 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 10 monclient(hunting): handle_auth_done global_id 5126 payload 274 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 1 -- 10.20.193.51:0/2603949535 >> v1:10.20.193.125:6789/0 conn(0x7fd4ac0c93d0 legacy=0x7fd4ac158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 1 -- 10.20.193.51:0/2603949535 >> v1:10.20.193.51:6789/0 conn(0x7fd4ac152ab0 legacy=0x7fd4ac0c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 1 monclient: found mon.noname-c 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.466+0000 7fd4b1761640 1 -- 10.20.193.51:0/2603949535 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac06e130 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.472 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.468+0000 7fd4b1761640 1 --2- 10.20.193.51:0/2603949535 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac0c8e90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00098d0 tx=0x7fd4a002ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a6299c9754d888 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 1 -- 10.20.193.51:0/2603949535 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a003c070 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.479 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: dump: 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.480 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 1 -- 10.20.193.51:0/2603949535 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a002fbf0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 5 monclient: authenticate success, global_id 5126 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 1 -- 10.20.193.51:0/2603949535 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0037440 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 4 set_mon_vals no callback set 2026-02-22T21:42:45.481 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b0f60640 10 monclient: dump: 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 10 monclient: shutdown 2026-02-22T21:42:45.482 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2603949535 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 msgr2=0x7fd4ac0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2603949535 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac0c8e90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00098d0 tx=0x7fd4a002ef20 comp rx=0 tx=0).stop 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2603949535 shutdown_connections 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2603949535 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac0c8e90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2603949535 >> 10.20.193.51:0/2603949535 conn(0x7fd4ac0cd7b0 msgr2=0x7fd4ac0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2603949535 shutdown_connections 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.470+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2603949535 wait complete. 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.483 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 10 monclient: monmap: 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 librados: starting msgr at 2026-02-22T21:42:45.484 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 librados: starting objecter 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 Processor -- start 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 -- start start 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 librados: setting wanted keys 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 1 librados: calling monclient init 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.471+0000 7fd4b41ed640 10 monclient: init 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac1486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-c con 0x7fd4ac06d9d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd4ac0c93d0 0x7fd4ac1f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-b con 0x7fd4ac0c93d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:45.485 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 0x7fd4ac1f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): picked mon.noname-a con 0x7fd4ac152ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac06e6f0 con 0x7fd4ac152ab0 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac0c9880 con 0x7fd4ac0c93d0 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac06e9d0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b41ed640 10 monclient(hunting): authenticate will time out at 1441.052120s 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 0x7fd4ac1f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac1486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 0x7fd4ac1f69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39532/0 (socket says 10.20.193.51:39532) 2026-02-22T21:42:45.486 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 1 -- 10.20.193.51:0/2231669570 learned_addr learned my addr 10.20.193.51:0/2231669570 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): get_auth_request con 0x7fd4ac152ab0 auth_method 0 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1761640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd4ac0c93d0 0x7fd4ac1f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): get_auth_request con 0x7fd4ac06d9d0 auth_method 0 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.487 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1761640 10 monclient(hunting): get_auth_request con 0x7fd4ac0c93d0 auth_method 0 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1761640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b1761640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.472+0000 7fd4b2763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.473+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.473+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.473+0000 7fd4b1761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.488 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 10 monclient(hunting): handle_auth_done global_id 5132 payload 824 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 -- 10.20.193.51:0/2231669570 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd4ac0c93d0 msgr2=0x7fd4ac1f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd4ac0c93d0 0x7fd4ac1f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 -- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 msgr2=0x7fd4ac1f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 0x7fd4ac1f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 monclient: found mon.noname-c 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 -- 10.20.193.51:0/2231669570 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac149260 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd4b1f62640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac1486f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c400 tx=0x7fd49c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.489 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd49a7fc640 10 client.?.objecter ms_handle_connect 0x7fd4ac06d9d0 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.474+0000 7fd49a7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c017070 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: dump: 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.490 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49c005de0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0072f0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49bfff640 4 set_mon_vals no callback set 2026-02-22T21:42:45.491 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd49a7fc640 10 monclient: dump: 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 5 monclient: authenticate success, global_id 5132 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 10 monclient: _renew_subs 2026-02-22T21:42:45.492 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac1f5110 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 10 client.5132.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 10 monclient: _renew_subs 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac1f56d0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.476+0000 7fd4b41ed640 1 librados: init done 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.477+0000 7fd4b41ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.477+0000 7fd4b41ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.477+0000 7fd4b41ed640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.493 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.477+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac152e90 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.488+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd49c007490 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.488+0000 7fd49a7fc640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 0x7fd47c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.489+0000 7fd4b1761640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 0x7fd47c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.489+0000 7fd4b1761640 10 monclient: get_auth_request con 0x7fd47c078e00 auth_method 0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.489+0000 7fd4b1761640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 0x7fd47c07b2c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0039000 tx=0x7fd4a0033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.489+0000 7fd49a7fc640 10 client.5132.objecter ms_handle_connect 0x7fd47c078e00 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.490+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd49c0077b0 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.490+0000 7fd49a7fc640 10 client.5132.objecter ms_dispatch2 0x7fd4ac000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:45.495 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.490+0000 7fd49a7fc640 3 client.5132.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:45.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.490+0000 7fd49a7fc640 3 client.5132.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:45.496 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.490+0000 7fd49a7fc640 20 client.5132.objecter dump_active .. 0 homeless 2026-02-22T21:42:45.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.590+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fd49c021020 con 0x7fd4ac06d9d0 2026-02-22T21:42:45.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.590+0000 7fd49a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.590+0000 7fd49a7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:45.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.673+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd4ac0c8d80 con 0x7fd47c078e00 2026-02-22T21:42:45.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.674+0000 7fd49a7fc640 1 -- 10.20.193.51:0/2231669570 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44291 (secure 0 0 0) 0x7fd4ac0c8d80 con 0x7fd47c078e00 2026-02-22T21:42:45.676 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 10 librados: watch_flush enter 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 10 librados: watch_flush exit 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 20 client.5132.objecter shutdown clearing up homeless session... 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 10 client.5132.objecter successfully canceled tick 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 msgr2=0x7fd47c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 0x7fd47c07b2c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0039000 tx=0x7fd4a0033000 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 10 monclient: shutdown 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 msgr2=0x7fd4ac1486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac1486f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c400 tx=0x7fd49c00c8d0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 shutdown_connections 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd47c078e00 0x7fd47c07b2c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd4ac152ab0 0x7fd4ac1f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd4ac0c93d0 0x7fd4ac1f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 --2- 10.20.193.51:0/2231669570 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd4ac06d9d0 0x7fd4ac1486f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 >> 10.20.193.51:0/2231669570 conn(0x7fd4ac0cd7b0 msgr2=0x7fd4ac1ea0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 shutdown_connections 2026-02-22T21:42:45.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 -- 10.20.193.51:0/2231669570 wait complete. 2026-02-22T21:42:45.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.675+0000 7fd4b41ed640 1 librados: shutdown 2026-02-22T21:42:45.685 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2026-02-22T21:42:43.765285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437528,"kb_used_data":6104,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509511912,"statfs":{"total":1546188226560,"available":1545740197888,"internally_reserved":0,"allocated":6250496,"data_stored":3091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001911"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:44.312096+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:39:44.312096+0000","last_peered":"2026-02-22T21:39:44.312096+0000","last_clean":"2026-02-22T21:39:44.312096+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:39:44.312096+0000","last_undegraded":"2026-02-22T21:39:44.312096+0000","last_fullsized":"2026-02-22T21:39:44.312096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:42.253559+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:39:42.253559+0000","last_peered":"2026-02-22T21:39:42.253559+0000","last_clean":"2026-02-22T21:39:42.253559+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:39:42.253559+0000","last_undegraded":"2026-02-22T21:39:42.253559+0000","last_fullsized":"2026-02-22T21:39:42.253559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:40.400160+0000","last_change":"2026-02-22T21:37:38.380932+0000","last_active":"2026-02-22T21:39:40.400160+0000","last_peered":"2026-02-22T21:39:40.400160+0000","last_clean":"2026-02-22T21:39:40.400160+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:39:40.400160+0000","last_undegraded":"2026-02-22T21:39:40.400160+0000","last_fullsized":"2026-02-22T21:39:40.400160+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T23:22:09.624552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:37.344398+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:39:37.344398+0000","last_peered":"2026-02-22T21:39:37.344398+0000","last_clean":"2026-02-22T21:39:37.344398+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:39:37.344398+0000","last_undegraded":"2026-02-22T21:39:37.344398+0000","last_fullsized":"2026-02-22T21:39:37.344398+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"30'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:39.310121+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:39:39.310121+0000","last_peered":"2026-02-22T21:39:39.310121+0000","last_clean":"2026-02-22T21:39:39.310121+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:39:39.310121+0000","last_undegraded":"2026-02-22T21:39:39.310121+0000","last_fullsized":"2026-02-22T21:39:39.310121+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437076+0000","last_change":"2026-02-22T21:37:36.829493+0000","last_active":"2026-02-22T21:39:38.437076+0000","last_peered":"2026-02-22T21:39:38.437076+0000","last_clean":"2026-02-22T21:39:38.437076+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:39:38.437076+0000","last_undegraded":"2026-02-22T21:39:38.437076+0000","last_fullsized":"2026-02-22T21:39:38.437076+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:59:05.328006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:33.547138+0000","last_change":"2026-02-22T21:37:38.428511+0000","last_active":"2026-02-22T21:39:33.547138+0000","last_peered":"2026-02-22T21:39:33.547138+0000","last_clean":"2026-02-22T21:39:33.547138+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:39:33.547138+0000","last_undegraded":"2026-02-22T21:39:33.547138+0000","last_fullsized":"2026-02-22T21:39:33.547138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:54:42.502365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:38.437292+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:39:38.437292+0000","last_peered":"2026-02-22T21:39:38.437292+0000","last_clean":"2026-02-22T21:39:38.437292+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:39:38.437292+0000","last_undegraded":"2026-02-22T21:39:38.437292+0000","last_fullsized":"2026-02-22T21:39:38.437292+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:04:41.120271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"30'32","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:28.703892+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:39:28.703892+0000","last_peered":"2026-02-22T21:39:28.703892+0000","last_clean":"2026-02-22T21:39:28.703892+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:39:28.703892+0000","last_undegraded":"2026-02-22T21:39:28.703892+0000","last_fullsized":"2026-02-22T21:39:28.703892+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313283,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:42:45.687 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 10 monclient: get_monmap_and_config 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 10 monclient: monmap: 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.756 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:45.755137+0000 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:45.755137+0000 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 1 Processor -- start 2026-02-22T21:42:45.757 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 1 -- start start 2026-02-22T21:42:45.758 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.754+0000 7f9e260bd640 10 monclient: init 2026-02-22T21:42:45.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient: _add_conns ranks=[4,2,5,1,0,3] 2026-02-22T21:42:45.760 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9e20152ab0 0x7f9e200c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9e20152ab0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e2006d9d0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f9e200c95a0 0x7f9e20158a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9e200c95a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2006f1d0 con 0x7f9e2006d9d0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f9e2006ed40 con 0x7f9e20152ab0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f9e2006de50 con 0x7f9e200c95a0 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e260bd640 10 monclient(hunting): authenticate will time out at 1441.335450s 2026-02-22T21:42:45.761 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.755+0000 7f9e1f7fe640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f9e20152ab0 0x7f9e200c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1effd640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f9e2006d9d0 0x7f9e200c8e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:46026/0 (socket says 10.20.193.51:46026) 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1effd640 1 -- 10.20.193.51:0/129161837 learned_addr learned my addr 10.20.193.51:0/129161837 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): get_auth_request con 0x7f9e20152ab0 auth_method 0 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 1 --2- 10.20.193.51:0/129161837 >> v2:10.20.193.125:3300/0 conn(0x7f9e200c95a0 0x7f9e20158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): get_auth_request con 0x7f9e200c95a0 auth_method 0 2026-02-22T21:42:45.762 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.756+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_done global_id 5158 payload 274 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 1 -- 10.20.193.51:0/129161837 >> v2:10.20.193.125:3300/0 conn(0x7f9e200c95a0 msgr2=0x7f9e20158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 1 --2- 10.20.193.51:0/129161837 >> v2:10.20.193.125:3300/0 conn(0x7f9e200c95a0 0x7f9e20158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 1 -- 10.20.193.51:0/129161837 >> v1:10.20.193.125:6789/0 conn(0x7f9e2006d9d0 legacy=0x7f9e200c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 1 monclient: found mon.noname-b 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.763 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:45.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.757+0000 7f9e1f7fe640 1 -- 10.20.193.51:0/129161837 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2006e130 con 0x7f9e20152ab0 2026-02-22T21:42:45.764 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.758+0000 7f9e1f7fe640 1 --2- 10.20.193.51:0/129161837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e20152ab0 0x7f9e200c5ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0098a0 tx=0x7f9e0c02eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a9d0e60afdd258 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 1 -- 10.20.193.51:0/129161837 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c03c070 con 0x7f9e20152ab0 2026-02-22T21:42:45.770 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: dump: 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.771 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 1 -- 10.20.193.51:0/129161837 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c02fb80 con 0x7f9e20152ab0 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 5 monclient: authenticate success, global_id 5158 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 1 -- 10.20.193.51:0/129161837 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c037440 con 0x7f9e20152ab0 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e1e7fc640 10 monclient: dump: 2026-02-22T21:42:45.772 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 4 set_mon_vals no callback set 2026-02-22T21:42:45.773 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 10 monclient: shutdown 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 -- 10.20.193.51:0/129161837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e20152ab0 msgr2=0x7f9e200c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 --2- 10.20.193.51:0/129161837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e20152ab0 0x7f9e200c5ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0098a0 tx=0x7f9e0c02eeb0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 -- 10.20.193.51:0/129161837 shutdown_connections 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 --2- 10.20.193.51:0/129161837 >> v2:10.20.193.125:3300/0 conn(0x7f9e200c95a0 0x7f9e20158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 --2- 10.20.193.51:0/129161837 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e20152ab0 0x7f9e200c5ab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 -- 10.20.193.51:0/129161837 >> 10.20.193.51:0/129161837 conn(0x7f9e200cd7b0 msgr2=0x7f9e200cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 -- 10.20.193.51:0/129161837 shutdown_connections 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.760+0000 7f9e260bd640 1 -- 10.20.193.51:0/129161837 wait complete. 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient: build_initial_monmap 2026-02-22T21:42:45.774 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient: monmap: 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 librados: starting msgr at 2026-02-22T21:42:45.775 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 librados: starting objecter 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 Processor -- start 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 -- start start 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 librados: setting wanted keys 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 librados: calling monclient init 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient: init 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e2006d9d0 0x7f9e201487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-a con 0x7f9e2006d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 0x7f9e201548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9e200c95a0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 0x7f9e20154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.776 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9e20152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2006e9d0 con 0x7f9e2006d9d0 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2006e6f0 con 0x7f9e200c95a0 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9e20152e90 con 0x7f9e20152ab0 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.761+0000 7f9e260bd640 10 monclient(hunting): authenticate will time out at 1441.341831s 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 0x7f9e201548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 0x7f9e20154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 0x7f9e201548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:58408/0 (socket says 10.20.193.51:58408) 2026-02-22T21:42:45.777 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 1 -- 10.20.193.51:0/3476410058 learned_addr learned my addr 10.20.193.51:0/3476410058 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): get_auth_request con 0x7f9e200c95a0 auth_method 0 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e2006d9d0 0x7f9e201487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): get_auth_request con 0x7f9e20152ab0 auth_method 0 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.778 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): get_auth_request con 0x7f9e2006d9d0 auth_method 0 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.762+0000 7f9e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 10 monclient(hunting): handle_auth_done global_id 5141 payload 824 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 -- 10.20.193.51:0/3476410058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 msgr2=0x7f9e201548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.779 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 0x7f9e201548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 -- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e2006d9d0 msgr2=0x7f9e201487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e2006d9d0 0x7f9e201487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 monclient: found mon.noname-c 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.763+0000 7f9e1ffff640 1 -- 10.20.193.51:0/3476410058 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e20153250 con 0x7f9e20152ab0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.764+0000 7f9e1ffff640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 0x7f9e20154df0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400ef90 tx=0x7f9e1400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.764+0000 7f9dfbfff640 10 client.?.objecter ms_handle_connect 0x7f9e20152ab0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.764+0000 7f9dfbfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14018020 con 0x7f9e20152ab0 2026-02-22T21:42:45.780 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: dump: 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.781 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: _finish_auth 0 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 5 monclient: authenticate success, global_id 5141 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e14014480 con 0x7f9e20152ab0 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14014870 con 0x7f9e20152ab0 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9dfbfff640 10 monclient: dump: 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.782 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e1dffb640 4 set_mon_vals no callback set 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _renew_subs 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e20153530 con 0x7f9e20152ab0 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 client.5141.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:45.783 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _renew_subs 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e20153af0 con 0x7f9e20152ab0 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 1 librados: init done 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.784 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.767+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e200c8e90 con 0x7f9e20152ab0 2026-02-22T21:42:45.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.873+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9e14014a10 con 0x7f9e20152ab0 2026-02-22T21:42:45.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 0x7f9df007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:45.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9e1f7fe640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 0x7f9df007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:45.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9e1f7fe640 10 monclient: get_auth_request con 0x7f9df0078e50 auth_method 0 2026-02-22T21:42:45.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9e1f7fe640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 0x7f9df007b310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c039000 tx=0x7f9e0c033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:45.876 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 10 client.5141.objecter ms_handle_connect 0x7f9df0078e50 2026-02-22T21:42:45.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9e1405eec0 con 0x7f9e20152ab0 2026-02-22T21:42:45.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 10 client.5141.objecter ms_dispatch2 0x7f9e20000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:45.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 3 client.5141.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:45.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.874+0000 7f9dfbfff640 3 client.5141.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:45.877 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.875+0000 7f9dfbfff640 20 client.5141.objecter dump_active .. 0 homeless 2026-02-22T21:42:45.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.882+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9e14022030 con 0x7f9e20152ab0 2026-02-22T21:42:45.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.882+0000 7f9dfbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:45.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.882+0000 7f9dfbfff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:45.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.967+0000 7f9e260bd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:42:45.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.967+0000 7f9e260bd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:42:45.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.967+0000 7f9e260bd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:45.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:45.967+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e20156380 con 0x7f9e20152ab0 2026-02-22T21:42:46.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.031+0000 7f9dfbfff640 1 -- 10.20.193.51:0/3476410058 <== mon.2 v2:10.20.193.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+24686 (secure 0 0 0) 0x7f9e14010040 con 0x7f9e20152ab0 2026-02-22T21:42:46.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.031+0000 7f9dfbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2026-02-22T21:42:46.034 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.031+0000 7f9dfbfff640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:42:46.034 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:46.035 INFO:teuthology.orchestra.run.trial051.stdout:{"epoch":31,"fsid":"f64be66c-6ce4-4d13-a1b8-cf6040bfed5c","created":"2026-02-22T21:35:46.835611+0000","modified":"2026-02-22T21:37:41.177631+0000","last_up_change":"2026-02-22T21:37:29.474866+0000","last_in_change":"2026-02-22T21:35:52.985822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T21:37:27.901086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-22T21:37:30.749692+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":30,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b8bc99a9-1c6a-4062-808d-e1fe0423f69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6802","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6803","nonce":1198482343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6804","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6805","nonce":1198482343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6808","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6809","nonce":1198482343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6806","nonce":1198482343},{"type":"v1","addr":"10.20.193.51:6807","nonce":1198482343}]},"public_addr":"10.20.193.51:6803/1198482343","cluster_addr":"10.20.193.51:6805/1198482343","heartbeat_back_addr":"10.20.193.51:6809/1198482343","heartbeat_front_addr":"10.20.193.51:6807/1198482343","state":["exists","up"]},{"osd":1,"uuid":"2f40aa3d-76d4-49e0-aadd-ac162c75d7d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6824","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6825","nonce":2540990948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6826","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6827","nonce":2540990948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6830","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6831","nonce":2540990948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6828","nonce":2540990948},{"type":"v1","addr":"10.20.193.68:6829","nonce":2540990948}]},"public_addr":"10.20.193.68:6825/2540990948","cluster_addr":"10.20.193.68:6827/2540990948","heartbeat_back_addr":"10.20.193.68:6831/2540990948","heartbeat_front_addr":"10.20.193.68:6829/2540990948","state":["exists","up"]},{"osd":2,"uuid":"fbb12d84-86d8-45a8-bbd9-23b4d202e27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6808","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6809","nonce":1963822926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6810","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6811","nonce":1963822926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6814","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6815","nonce":1963822926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6812","nonce":1963822926},{"type":"v1","addr":"10.20.193.125:6813","nonce":1963822926}]},"public_addr":"10.20.193.125:6809/1963822926","cluster_addr":"10.20.193.125:6811/1963822926","heartbeat_back_addr":"10.20.193.125:6815/1963822926","heartbeat_front_addr":"10.20.193.125:6813/1963822926","state":["exists","up"]},{"osd":3,"uuid":"d2387c43-87b8-4c8c-b5c7-9a6b7cf48bf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6824","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6825","nonce":3567184175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6826","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6827","nonce":3567184175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6830","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6831","nonce":3567184175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6828","nonce":3567184175},{"type":"v1","addr":"10.20.193.140:6829","nonce":3567184175}]},"public_addr":"10.20.193.140:6825/3567184175","cluster_addr":"10.20.193.140:6827/3567184175","heartbeat_back_addr":"10.20.193.140:6831/3567184175","heartbeat_front_addr":"10.20.193.140:6829/3567184175","state":["exists","up"]},{"osd":4,"uuid":"6679621e-8d0d-405b-b6db-ef0d22409ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6810","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6811","nonce":1021106712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6812","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6813","nonce":1021106712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6816","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6817","nonce":1021106712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6814","nonce":1021106712},{"type":"v1","addr":"10.20.193.51:6815","nonce":1021106712}]},"public_addr":"10.20.193.51:6811/1021106712","cluster_addr":"10.20.193.51:6813/1021106712","heartbeat_back_addr":"10.20.193.51:6817/1021106712","heartbeat_front_addr":"10.20.193.51:6815/1021106712","state":["exists","up"]},{"osd":5,"uuid":"c044485c-e401-4a33-885e-8f573934ea0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6816","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6817","nonce":3736303751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6818","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6819","nonce":3736303751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6822","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6823","nonce":3736303751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6820","nonce":3736303751},{"type":"v1","addr":"10.20.193.68:6821","nonce":3736303751}]},"public_addr":"10.20.193.68:6817/3736303751","cluster_addr":"10.20.193.68:6819/3736303751","heartbeat_back_addr":"10.20.193.68:6823/3736303751","heartbeat_front_addr":"10.20.193.68:6821/3736303751","state":["exists","up"]},{"osd":6,"uuid":"6908a0e5-ffe4-4cb7-a0ad-43053ea716b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6824","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6825","nonce":1184344866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6826","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6827","nonce":1184344866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6830","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6831","nonce":1184344866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6828","nonce":1184344866},{"type":"v1","addr":"10.20.193.125:6829","nonce":1184344866}]},"public_addr":"10.20.193.125:6825/1184344866","cluster_addr":"10.20.193.125:6827/1184344866","heartbeat_back_addr":"10.20.193.125:6831/1184344866","heartbeat_front_addr":"10.20.193.125:6829/1184344866","state":["exists","up"]},{"osd":7,"uuid":"97cb63ed-8a06-4d76-ac73-f8ef1b5cf1d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6808","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6809","nonce":1140578699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6810","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6811","nonce":1140578699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6814","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6815","nonce":1140578699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6812","nonce":1140578699},{"type":"v1","addr":"10.20.193.140:6813","nonce":1140578699}]},"public_addr":"10.20.193.140:6809/1140578699","cluster_addr":"10.20.193.140:6811/1140578699","heartbeat_back_addr":"10.20.193.140:6815/1140578699","heartbeat_front_addr":"10.20.193.140:6813/1140578699","state":["exists","up"]},{"osd":8,"uuid":"68963ece-62bb-468d-83aa-461d5d15000d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6826","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6827","nonce":142198174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6828","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6829","nonce":142198174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6832","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6833","nonce":142198174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6830","nonce":142198174},{"type":"v1","addr":"10.20.193.51:6831","nonce":142198174}]},"public_addr":"10.20.193.51:6827/142198174","cluster_addr":"10.20.193.51:6829/142198174","heartbeat_back_addr":"10.20.193.51:6833/142198174","heartbeat_front_addr":"10.20.193.51:6831/142198174","state":["exists","up"]},{"osd":9,"uuid":"6b0f80c5-bcc0-4c72-84d6-af143276a0cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6808","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6809","nonce":1841964582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6810","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6811","nonce":1841964582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6814","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6815","nonce":1841964582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6812","nonce":1841964582},{"type":"v1","addr":"10.20.193.68:6813","nonce":1841964582}]},"public_addr":"10.20.193.68:6809/1841964582","cluster_addr":"10.20.193.68:6811/1841964582","heartbeat_back_addr":"10.20.193.68:6815/1841964582","heartbeat_front_addr":"10.20.193.68:6813/1841964582","state":["exists","up"]},{"osd":10,"uuid":"7c884a35-46d8-45ae-b417-ddb6d976526b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6800","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6801","nonce":1951583296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6802","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6803","nonce":1951583296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6806","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6807","nonce":1951583296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6804","nonce":1951583296},{"type":"v1","addr":"10.20.193.125:6805","nonce":1951583296}]},"public_addr":"10.20.193.125:6801/1951583296","cluster_addr":"10.20.193.125:6803/1951583296","heartbeat_back_addr":"10.20.193.125:6807/1951583296","heartbeat_front_addr":"10.20.193.125:6805/1951583296","state":["exists","up"]},{"osd":11,"uuid":"9e72c740-4dd0-490b-bad9-483da990d334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6816","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6817","nonce":2298497951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6818","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6819","nonce":2298497951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6822","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6823","nonce":2298497951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6820","nonce":2298497951},{"type":"v1","addr":"10.20.193.140:6821","nonce":2298497951}]},"public_addr":"10.20.193.140:6817/2298497951","cluster_addr":"10.20.193.140:6819/2298497951","heartbeat_back_addr":"10.20.193.140:6823/2298497951","heartbeat_front_addr":"10.20.193.140:6821/2298497951","state":["exists","up"]},{"osd":12,"uuid":"e85b2358-a072-4155-a38f-68f53529ebe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6818","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6819","nonce":1676524147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6820","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6821","nonce":1676524147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6824","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6825","nonce":1676524147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.51:6822","nonce":1676524147},{"type":"v1","addr":"10.20.193.51:6823","nonce":1676524147}]},"public_addr":"10.20.193.51:6819/1676524147","cluster_addr":"10.20.193.51:6821/1676524147","heartbeat_back_addr":"10.20.193.51:6825/1676524147","heartbeat_front_addr":"10.20.193.51:6823/1676524147","state":["exists","up"]},{"osd":13,"uuid":"ca68e92b-99ca-4584-825d-9c0c601134c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6800","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6801","nonce":762840177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6802","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6803","nonce":762840177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6806","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6807","nonce":762840177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.68:6804","nonce":762840177},{"type":"v1","addr":"10.20.193.68:6805","nonce":762840177}]},"public_addr":"10.20.193.68:6801/762840177","cluster_addr":"10.20.193.68:6803/762840177","heartbeat_back_addr":"10.20.193.68:6807/762840177","heartbeat_front_addr":"10.20.193.68:6805/762840177","state":["exists","up"]},{"osd":14,"uuid":"ce0888c2-dcdf-44c5-a8d4-06a1cd783264","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6816","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6817","nonce":498199928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6818","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6819","nonce":498199928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6822","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6823","nonce":498199928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.125:6820","nonce":498199928},{"type":"v1","addr":"10.20.193.125:6821","nonce":498199928}]},"public_addr":"10.20.193.125:6817/498199928","cluster_addr":"10.20.193.125:6819/498199928","heartbeat_back_addr":"10.20.193.125:6823/498199928","heartbeat_front_addr":"10.20.193.125:6821/498199928","state":["exists","up"]},{"osd":15,"uuid":"44725141-dc07-4bce-b2c5-121a3c85c67e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6800","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6801","nonce":2657947218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6802","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6803","nonce":2657947218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6806","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6807","nonce":2657947218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.140:6804","nonce":2657947218},{"type":"v1","addr":"10.20.193.140:6805","nonce":2657947218}]},"public_addr":"10.20.193.140:6801/2657947218","cluster_addr":"10.20.193.140:6803/2657947218","heartbeat_back_addr":"10.20.193.140:6807/2657947218","heartbeat_front_addr":"10.20.193.140:6805/2657947218","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.521455+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.773204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.592098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.942138+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.663559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.647770+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.811906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.107957+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.279951+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.927867+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.306670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:25.794057+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:27.055278+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.699761+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:26.177807+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T21:37:24.554387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T21:42:46.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 10 librados: watch_flush enter 2026-02-22T21:42:46.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 10 librados: watch_flush exit 2026-02-22T21:42:46.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 20 client.5141.objecter shutdown clearing up homeless session... 2026-02-22T21:42:46.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 10 client.5141.objecter successfully canceled tick 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 msgr2=0x7f9df007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 0x7f9df007b310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c039000 tx=0x7f9e0c033000 comp rx=0 tx=0).stop 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 10 monclient: shutdown 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 msgr2=0x7f9e20154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 0x7f9e20154df0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400ef90 tx=0x7f9e1400c490 comp rx=0 tx=0).stop 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 shutdown_connections 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9df0078e50 0x7f9df007b310 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9e20152ab0 0x7f9e20154df0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9e200c95a0 0x7f9e201548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:46.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 --2- 10.20.193.51:0/3476410058 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9e2006d9d0 0x7f9e201487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:46.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 >> 10.20.193.51:0/3476410058 conn(0x7f9e200cd7b0 msgr2=0x7f9e200cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:46.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 shutdown_connections 2026-02-22T21:42:46.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 -- 10.20.193.51:0/3476410058 wait complete. 2026-02-22T21:42:46.038 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:46.033+0000 7f9e260bd640 1 librados: shutdown 2026-02-22T21:42:47.045 INFO:tasks.ceph:Scrubbing osd.0 2026-02-22T21:42:47.045 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:47.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:47.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 10 monclient: monmap: 2026-02-22T21:42:47.116 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:47.116067+0000 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:47.116067+0000 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 1 Processor -- start 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 1 -- start start 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.115+0000 7f038d52c640 10 monclient: init 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient: _add_conns ranks=[3,1,5,2,4,0] 2026-02-22T21:42:47.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0388155c40 0x7f03880c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-a con 0x7f0388155c40 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f03880c8190 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f0388073b50 0x7f038815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-c con 0x7f0388073b50 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03880c8620 con 0x7f03880c8190 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f03880c8900 con 0x7f0388155c40 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f03880c8be0 con 0x7f0388073b50 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f038d52c640 10 monclient(hunting): authenticate will time out at 1442.696479s 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f0386ffd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f0388155c40 0x7f03880c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f03877fe640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f0388073b50 0x7f038815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f03867fc640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f03880c8190 0x7f0388073440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:36354/0 (socket says 10.20.193.51:36354) 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f03867fc640 1 -- 10.20.193.51:0/3046100411 learned_addr learned my addr 10.20.193.51:0/3046100411 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f0386ffd640 10 monclient(hunting): get_auth_request con 0x7f0388155c40 auth_method 0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f0386ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f0386ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.116+0000 7f0386ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): get_auth_request con 0x7f0388073b50 auth_method 0 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.117+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 10 monclient(hunting): handle_auth_done global_id 5169 payload 274 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 1 -- 10.20.193.51:0/3046100411 >> v2:10.20.193.125:3300/0 conn(0x7f0388073b50 msgr2=0x7f038815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 1 --2- 10.20.193.51:0/3046100411 >> v2:10.20.193.125:3300/0 conn(0x7f0388073b50 0x7f038815d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 1 -- 10.20.193.51:0/3046100411 >> v1:10.20.193.68:6789/0 conn(0x7f03880c8190 legacy=0x7f0388073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 1 monclient: found mon.noname-a 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:47.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.118+0000 7f0386ffd640 1 -- 10.20.193.51:0/3046100411 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03880c8ec0 con 0x7f0388155c40 2026-02-22T21:42:47.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.119+0000 7f0386ffd640 1 --2- 10.20.193.51:0/3046100411 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388155c40 0x7f03880c7c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0388160340 tx=0x7f037c02ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191cf1c912181a83 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 1 -- 10.20.193.51:0/3046100411 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c03c070 con 0x7f0388155c40 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: dump: 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 1 -- 10.20.193.51:0/3046100411 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c02faf0 con 0x7f0388155c40 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f038d52c640 5 monclient: authenticate success, global_id 5169 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f038d52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 1 -- 10.20.193.51:0/3046100411 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c037440 con 0x7f0388155c40 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f038d52c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f038d52c640 4 set_mon_vals no callback set 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.120+0000 7f0385ffb640 10 monclient: dump: 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.123 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 10 monclient: shutdown 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- 10.20.193.51:0/3046100411 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388155c40 msgr2=0x7f03880c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 --2- 10.20.193.51:0/3046100411 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388155c40 0x7f03880c7c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0388160340 tx=0x7f037c02ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- 10.20.193.51:0/3046100411 shutdown_connections 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 --2- 10.20.193.51:0/3046100411 >> v2:10.20.193.125:3300/0 conn(0x7f0388073b50 0x7f038815d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 --2- 10.20.193.51:0/3046100411 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388155c40 0x7f03880c7c50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- 10.20.193.51:0/3046100411 >> 10.20.193.51:0/3046100411 conn(0x7f03880c9d50 msgr2=0x7f03880ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- 10.20.193.51:0/3046100411 shutdown_connections 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- 10.20.193.51:0/3046100411 wait complete. 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 10 monclient: monmap: 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:47.125 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 librados: starting msgr at 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 librados: starting objecter 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 Processor -- start 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 -- start start 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 librados: setting wanted keys 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 1 librados: calling monclient init 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.121+0000 7f038d52c640 10 monclient: init 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 0x7f038815b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-a con 0x7f0388073b50 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0388155c40 0x7f03881587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-b con 0x7f0388155c40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f038815bac0 0x7f0388158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): picked mon.noname-c con 0x7f038815bac0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f0388156020 con 0x7f0388073b50 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f03880c9480 con 0x7f0388155c40 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f03881561a0 con 0x7f038815bac0 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f038d52c640 10 monclient(hunting): authenticate will time out at 1442.702334s 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 0x7f038815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 0x7f038815b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:39566/0 (socket says 10.20.193.51:39566) 2026-02-22T21:42:47.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 1 -- 10.20.193.51:0/960431049 learned_addr learned my addr 10.20.193.51:0/960431049 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): get_auth_request con 0x7f0388073b50 auth_method 0 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03877fe640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f038815bac0 0x7f0388158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03867fc640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0388155c40 0x7f03881587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03877fe640 10 monclient(hunting): get_auth_request con 0x7f038815bac0 auth_method 0 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03877fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03877fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03867fc640 10 monclient(hunting): get_auth_request con 0x7f0388155c40 auth_method 0 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03867fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f03867fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.122+0000 7f0386ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.123+0000 7f03867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 10 monclient(hunting): handle_auth_done global_id 5172 payload 824 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 -- 10.20.193.51:0/960431049 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f038815bac0 msgr2=0x7f0388158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f038815bac0 0x7f0388158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 -- 10.20.193.51:0/960431049 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0388155c40 msgr2=0x7f03881587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0388155c40 0x7f03881587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 monclient: found mon.noname-a 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.124+0000 7f0386ffd640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03881f3a00 con 0x7f0388073b50 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.125+0000 7f0386ffd640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 0x7f038815b580 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f037c009130 tx=0x7f037c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.125+0000 7f03677fe640 10 client.?.objecter ms_handle_connect 0x7f0388073b50 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.125+0000 7f03677fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c03c040 con 0x7f0388073b50 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: dump: 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c042610 con 0x7f0388073b50 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c0429e0 con 0x7f0388073b50 2026-02-22T21:42:47.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03677fe640 10 monclient: dump: 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f03857fa640 4 set_mon_vals no callback set 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 5 monclient: authenticate success, global_id 5172 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 10 monclient: _renew_subs 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881f3ce0 con 0x7f0388073b50 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 10 client.5172.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 10 monclient: _renew_subs 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881f42a0 con 0x7f0388073b50 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.126+0000 7f038d52c640 1 librados: init done 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 10 client.5172.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 15 client.5172.objecter _session_command_op_assign -1 1 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 20 client.5172.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 10 client.5172.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:47.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.127+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f03880c7c50 con 0x7f0388073b50 2026-02-22T21:42:47.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.138+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f037c04b450 con 0x7f0388073b50 2026-02-22T21:42:47.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.139+0000 7f03677fe640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0354078e50 0x7f035407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.139+0000 7f03867fc640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0354078e50 0x7f035407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.139+0000 7f03867fc640 10 monclient: get_auth_request con 0x7f0354078e50 auth_method 0 2026-02-22T21:42:47.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.139+0000 7f03677fe640 10 client.5172.objecter ms_handle_connect 0x7f0354078e50 2026-02-22T21:42:47.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.139+0000 7f03867fc640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0354078e50 0x7f035407b310 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0388150990 tx=0x7f0370009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f037c02fa70 con 0x7f0388073b50 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 10 client.5172.objecter ms_dispatch2 0x7f0388000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 3 client.5172.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 3 client.5172.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 10 client.5172.objecter checking command 1 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7f0354081330 0x7f0354083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter _get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 15 client.5172.objecter _session_command_op_remove -1 1 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter _get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 15 client.5172.objecter _session_command_op_assign 0 1 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 4 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 10 client.5172.objecter _send_command 1 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f037c039e90 con 0x7f0354081330 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.140+0000 7f03677fe640 20 client.5172.objecter dump_active .. 0 homeless 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.141+0000 7f03877fe640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7f0354081330 0x7f0354083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.141+0000 7f03877fe640 10 monclient: get_auth_request con 0x7f0354081330 auth_method 0 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.141+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f037c082d50 con 0x7f0388073b50 2026-02-22T21:42:47.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.141+0000 7f03677fe640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:47.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.143+0000 7f03677fe640 10 client.5172.objecter ms_handle_connect 0x7f0354081330 2026-02-22T21:42:47.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.143+0000 7f03877fe640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7f0354081330 0x7f0354083750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.152+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== osd.0 v2:10.20.193.51:6802/1198482343 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f037c039e90 con 0x7f0354081330 2026-02-22T21:42:47.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.152+0000 7f03677fe640 10 client.5172.objecter ms_dispatch2 0x7f0388000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:47.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.152+0000 7f03677fe640 10 client.5172.objecter _finish_command 1 = system:0 2026-02-22T21:42:47.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.152+0000 7f03677fe640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 4 2026-02-22T21:42:47.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.152+0000 7f03677fe640 15 client.5172.objecter _session_command_op_remove 0 1 2026-02-22T21:42:47.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 10 client.5172.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:47.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 15 client.5172.objecter _session_command_op_assign -1 2 2026-02-22T21:42:47.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 20 client.5172.objecter _get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 20 client.5172.objecter _get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 15 client.5172.objecter _session_command_op_remove -1 2 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 20 client.5172.objecter get_session s=0x7f0354080c60 osd=0 3 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 15 client.5172.objecter _session_command_op_assign 0 2 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 4 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 10 client.5172.objecter _send_command 2 2026-02-22T21:42:47.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.158+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 --> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0388152c70 con 0x7f0354081330 2026-02-22T21:42:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03677fe640 1 -- 10.20.193.51:0/960431049 <== osd.0 v2:10.20.193.51:6802/1198482343 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f0388152c70 con 0x7f0354081330 2026-02-22T21:42:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03677fe640 10 client.5172.objecter ms_dispatch2 0x7f0388000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03677fe640 10 client.5172.objecter _finish_command 2 = system:0 2026-02-22T21:42:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03677fe640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 4 2026-02-22T21:42:47.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03677fe640 15 client.5172.objecter _session_command_op_remove 0 2 2026-02-22T21:42:47.175 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:47.175 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:47.175 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 10 librados: watch_flush enter 2026-02-22T21:42:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 10 librados: watch_flush exit 2026-02-22T21:42:47.176 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 10 client.5172.objecter close_session for osd.0 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7f0354081330 msgr2=0x7f0354083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6802/1198482343,v1:10.20.193.51:6803/1198482343] conn(0x7f0354081330 0x7f0354083750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 20 client.5172.objecter put_session s=0x7f0354080c60 osd=0 1 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 20 client.5172.objecter shutdown clearing up homeless session... 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 10 client.5172.objecter successfully canceled tick 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0354078e50 msgr2=0x7f035407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0354078e50 0x7f035407b310 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0388150990 tx=0x7f0370009040 comp rx=0 tx=0).stop 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 10 monclient: shutdown 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 msgr2=0x7f038815b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 --2- 10.20.193.51:0/960431049 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0388073b50 0x7f038815b580 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f037c009130 tx=0x7f037c0047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f03877fe640 1 -- 10.20.193.51:0/960431049 reap_dead start 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 shutdown_connections 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.172+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 >> 10.20.193.51:0/960431049 conn(0x7f03880c9d50 msgr2=0x7f0388153470 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.173+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 shutdown_connections 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.173+0000 7f038d52c640 1 -- 10.20.193.51:0/960431049 wait complete. 2026-02-22T21:42:47.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.173+0000 7f038d52c640 1 librados: shutdown 2026-02-22T21:42:47.182 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-22T21:42:47.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 10 monclient: get_monmap_and_config 2026-02-22T21:42:47.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 10 monclient: monmap: 2026-02-22T21:42:47.251 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.251 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:47.250106+0000 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:47.250106+0000 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:47.252 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 1 Processor -- start 2026-02-22T21:42:47.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 1 -- start start 2026-02-22T21:42:47.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.249+0000 7fdce2a06640 10 monclient: init 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient: _add_conns ranks=[2,1,5,3,4,0] 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcdc150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fdcdc15c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdcdc0703b0 0x7fdcdc0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc0703b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcdc15a920 con 0x7fdcdc15c710 2026-02-22T21:42:47.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcdc15af60 con 0x7fdcdc150220 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fdcdc15b240 con 0x7fdcdc0703b0 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce2a06640 10 monclient(hunting): authenticate will time out at 1442.830434s 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.250+0000 7fdce0f7c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fdcdc0703b0 0x7fdcdc0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdcdb7fe640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fdcdc15c710 0x7fdcdc06fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:38938/0 (socket says 10.20.193.51:38938) 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdcdb7fe640 1 -- 10.20.193.51:0/1253528781 learned_addr learned my addr 10.20.193.51:0/1253528781 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): get_auth_request con 0x7fdcdc0703b0 auth_method 0 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.251+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_done global_id 5150 payload 274 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 1 -- 10.20.193.51:0/1253528781 >> v1:10.20.193.125:6789/0 conn(0x7fdcdc150220 legacy=0x7fdcdc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 1 -- 10.20.193.51:0/1253528781 >> v1:10.20.193.68:6789/0 conn(0x7fdcdc15c710 legacy=0x7fdcdc06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.252+0000 7fdce0f7c640 1 -- 10.20.193.51:0/1253528781 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc15b520 con 0x7fdcdc0703b0 2026-02-22T21:42:47.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.253+0000 7fdce0f7c640 1 --2- 10.20.193.51:0/1253528781 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc0c73a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0009dc0 tx=0x7fdcd002f2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9cd744ad7e9d10d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 1 -- 10.20.193.51:0/1253528781 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd003c070 con 0x7fdcdc0703b0 2026-02-22T21:42:47.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: dump: 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.265 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdce2a06640 5 monclient: authenticate success, global_id 5150 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 1 -- 10.20.193.51:0/1253528781 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd0037440 con 0x7fdcdc0703b0 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdce2a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 1 -- 10.20.193.51:0/1253528781 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0037810 con 0x7fdcdc0703b0 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdce2a06640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdce2a06640 4 set_mon_vals no callback set 2026-02-22T21:42:47.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.255+0000 7fdcdaffd640 10 monclient: dump: 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.267 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 10 monclient: shutdown 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 -- 10.20.193.51:0/1253528781 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc0703b0 msgr2=0x7fdcdc0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 --2- 10.20.193.51:0/1253528781 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc0c73a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0009dc0 tx=0x7fdcd002f2a0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 -- 10.20.193.51:0/1253528781 shutdown_connections 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 --2- 10.20.193.51:0/1253528781 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc0c73a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 -- 10.20.193.51:0/1253528781 >> 10.20.193.51:0/1253528781 conn(0x7fdcdc0cd7b0 msgr2=0x7fdcdc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 -- 10.20.193.51:0/1253528781 shutdown_connections 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 1 -- 10.20.193.51:0/1253528781 wait complete. 2026-02-22T21:42:47.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.256+0000 7fdce2a06640 10 monclient: monmap: 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 librados: starting msgr at 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 librados: starting objecter 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 Processor -- start 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 -- start start 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 librados: setting wanted keys 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 librados: calling monclient init 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient: init 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-b con 0x7fdcdc0703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 0x7fdcdc14cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-a con 0x7fdcdc150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 0x7fdcdc1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc150600 con 0x7fdcdc150220 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc15bae0 con 0x7fdcdc0703b0 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc150780 con 0x7fdcdc15c710 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce2a06640 10 monclient(hunting): authenticate will time out at 1442.837598s 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdcdb7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 0x7fdcdc14cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.257+0000 7fdce0f7c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 0x7fdcdc1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 0x7fdcdc14cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35288/0 (socket says 10.20.193.51:35288) 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 1 -- 10.20.193.51:0/2537115100 learned_addr learned my addr 10.20.193.51:0/2537115100 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): get_auth_request con 0x7fdcdc150220 auth_method 0 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): get_auth_request con 0x7fdcdc15c710 auth_method 0 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): get_auth_request con 0x7fdcdc0703b0 auth_method 0 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.258+0000 7fdcdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 10 monclient(hunting): handle_auth_done global_id 5153 payload 824 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 -- 10.20.193.51:0/2537115100 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdcdc0703b0 msgr2=0x7fdcdc154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 -- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 msgr2=0x7fdcdc14cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 0x7fdcdc14cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.259+0000 7fdce0f7c640 1 -- 10.20.193.51:0/2537115100 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc154ac0 con 0x7fdcdc15c710 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.260+0000 7fdce0f7c640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 0x7fdcdc1560a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.260+0000 7fdcd8ff9640 10 client.?.objecter ms_handle_connect 0x7fdcdc15c710 2026-02-22T21:42:47.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.260+0000 7fdcd8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd003c050 con 0x7fdcdc15c710 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: dump: 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.275 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd0048070 con 0x7fdcdc15c710 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0035d80 con 0x7fdcdc15c710 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcda7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdcd8ff9640 10 monclient: dump: 2026-02-22T21:42:47.276 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 5 monclient: authenticate success, global_id 5153 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _renew_subs 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc1f81c0 con 0x7fdcdc15c710 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 client.5153.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _renew_subs 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcdc0beaf0 con 0x7fdcdc15c710 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 1 librados: init done 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.262+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcdc0c8520 con 0x7fdcdc15c710 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.274+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fdcd004d460 con 0x7fdcdc15c710 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.275+0000 7fdcd8ff9640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 0x7fdcac07b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.275+0000 7fdcdbfff640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 0x7fdcac07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.275+0000 7fdcdbfff640 10 monclient: get_auth_request con 0x7fdcac078e80 auth_method 0 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.275+0000 7fdcd8ff9640 10 client.5153.objecter ms_handle_connect 0x7fdcac078e80 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.275+0000 7fdcdbfff640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 0x7fdcac07b340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005160 tx=0x7fdcc8007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.276+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fdcd00845c0 con 0x7fdcdc15c710 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.276+0000 7fdcd8ff9640 10 client.5153.objecter ms_dispatch2 0x7fdcdc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.276+0000 7fdcd8ff9640 3 client.5153.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.276+0000 7fdcd8ff9640 3 client.5153.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:47.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.276+0000 7fdcd8ff9640 20 client.5153.objecter dump_active .. 0 homeless 2026-02-22T21:42:47.378 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.377+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fdcd0088580 con 0x7fdcdc15c710 2026-02-22T21:42:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.377+0000 7fdcd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.379 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.377+0000 7fdcd8ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:47.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.459+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fdcdc074da0 con 0x7fdcac078e80 2026-02-22T21:42:47.461 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.459+0000 7fdcd8ff9640 1 -- 10.20.193.51:0/2537115100 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdcdc074da0 con 0x7fdcac078e80 2026-02-22T21:42:47.461 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 10 librados: watch_flush enter 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 10 librados: watch_flush exit 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 20 client.5153.objecter shutdown clearing up homeless session... 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 10 client.5153.objecter successfully canceled tick 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 msgr2=0x7fdcac07b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 0x7fdcac07b340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005160 tx=0x7fdcc8007f30 comp rx=0 tx=0).stop 2026-02-22T21:42:47.463 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 10 monclient: shutdown 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 msgr2=0x7fdcdc1560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 0x7fdcdc1560a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 shutdown_connections 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdcac078e80 0x7fdcac07b340 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdcdc15c710 0x7fdcdc1560a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdcdc150220 0x7fdcdc14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 --2- 10.20.193.51:0/2537115100 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdcdc0703b0 0x7fdcdc154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 >> 10.20.193.51:0/2537115100 conn(0x7fdcdc0cd7b0 msgr2=0x7fdcdc15c000 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 shutdown_connections 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 -- 10.20.193.51:0/2537115100 wait complete. 2026-02-22T21:42:47.464 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.460+0000 7fdce2a06640 1 librados: shutdown 2026-02-22T21:42:47.470 INFO:tasks.ceph:Scrubbing osd.4 2026-02-22T21:42:47.470 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:47.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 10 monclient: get_monmap_and_config 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 10 monclient: monmap: 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:47.539908+0000 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:47.539908+0000 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:47.541 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:47.542 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:47.542 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:47.542 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:47.542 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 1 Processor -- start 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 1 -- start start 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.539+0000 7fd6fdb1b640 10 monclient: init 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient: _add_conns ranks=[2,4,1,0,5,3] 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6f8153860 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fd6f80c9500 0x7fd6f815d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f80c9500 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fd6f80c7c50 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f80c8180 con 0x7fd6f80c7c50 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f80c8460 con 0x7fd6f8153860 2026-02-22T21:42:47.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f80c8740 con 0x7fd6f80c9500 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6fdb1b640 10 monclient(hunting): authenticate will time out at 1443.120312s 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6f77fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fd6f8153860 0x7fd6f815af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47530/0 (socket says 10.20.193.51:47530) 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6f77fe640 1 -- 10.20.193.51:0/2265296790 learned_addr learned my addr 10.20.193.51:0/2265296790 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.540+0000 7fd6f6ffd640 1 --2- 10.20.193.51:0/2265296790 >> v2:10.20.193.68:3300/0 conn(0x7fd6f80c9500 0x7fd6f815d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6f80c9500 auth_method 0 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 1 -- 10.20.193.51:0/2265296790 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154755412 0 0) 0x7fd6f80c8460 con 0x7fd6f8153860 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 10 monclient(hunting): _negotiate 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 10 monclient(hunting): my global_id is 5156 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.541+0000 7fd6f67fc640 1 -- 10.20.193.51:0/2265296790 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6dc003620 con 0x7fd6f8153860 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_done global_id 5176 payload 274 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 1 -- 10.20.193.51:0/2265296790 >> v1:10.20.193.125:6789/0 conn(0x7fd6f8153860 legacy=0x7fd6f815af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 1 -- 10.20.193.51:0/2265296790 >> v1:10.20.193.68:6789/0 conn(0x7fd6f80c7c50 legacy=0x7fd6f80c4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 1 monclient: found mon.noname-b 2026-02-22T21:42:47.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:47.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.542+0000 7fd6f6ffd640 1 -- 10.20.193.51:0/2265296790 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80c8a20 con 0x7fd6f80c9500 2026-02-22T21:42:47.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.544+0000 7fd6f6ffd640 1 --2- 10.20.193.51:0/2265296790 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c9500 0x7fd6f815d010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80098a0 tx=0x7fd6e802ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499d5268c8a9700 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 1 -- 10.20.193.51:0/2265296790 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e803c070 con 0x7fd6f80c9500 2026-02-22T21:42:47.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: dump: 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.554 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 1 -- 10.20.193.51:0/2265296790 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e802fb70 con 0x7fd6f80c9500 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6fdb1b640 5 monclient: authenticate success, global_id 5176 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 1 -- 10.20.193.51:0/2265296790 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8037440 con 0x7fd6f80c9500 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6fdb1b640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6fdb1b640 4 set_mon_vals no callback set 2026-02-22T21:42:47.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.546+0000 7fd6f67fc640 10 monclient: dump: 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 10 monclient: shutdown 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/2265296790 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c9500 msgr2=0x7fd6f815d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 --2- 10.20.193.51:0/2265296790 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c9500 0x7fd6f815d010 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80098a0 tx=0x7fd6e802ee60 comp rx=0 tx=0).stop 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/2265296790 shutdown_connections 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 --2- 10.20.193.51:0/2265296790 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c9500 0x7fd6f815d010 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/2265296790 >> 10.20.193.51:0/2265296790 conn(0x7fd6f80c9d50 msgr2=0x7fd6f80ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/2265296790 shutdown_connections 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/2265296790 wait complete. 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 10 monclient: monmap: 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.557 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 librados: starting msgr at 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 librados: starting objecter 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 Processor -- start 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 -- start start 2026-02-22T21:42:47.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 librados: setting wanted keys 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 1 librados: calling monclient init 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.547+0000 7fd6fdb1b640 10 monclient: init 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 0x7fd6f81f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f80c7c50 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd6f8153860 0x7fd6f814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f8153860 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd6f815d580 0x7fd6f81f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f815d580 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8153c40 con 0x7fd6f8153860 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f80c8fe0 con 0x7fd6f80c7c50 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8153dc0 con 0x7fd6f815d580 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6fdb1b640 10 monclient(hunting): authenticate will time out at 1443.128370s 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 0x7fd6f81f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f6ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd6f8153860 0x7fd6f814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 0x7fd6f81f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:40898/0 (socket says 10.20.193.51:40898) 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 1 -- 10.20.193.51:0/1210463487 learned_addr learned my addr 10.20.193.51:0/1210463487 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 10 monclient(hunting): get_auth_request con 0x7fd6f80c7c50 auth_method 0 2026-02-22T21:42:47.560 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f7fff640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd6f815d580 0x7fd6f81f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6f8153860 auth_method 0 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.548+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): get_auth_request con 0x7fd6f815d580 auth_method 0 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.549+0000 7fd6f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 10 monclient(hunting): handle_auth_done global_id 5182 payload 824 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 -- 10.20.193.51:0/1210463487 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd6f815d580 msgr2=0x7fd6f81f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fd6f815d580 0x7fd6f81f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 -- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd6f8153860 msgr2=0x7fd6f814db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fd6f8153860 0x7fd6f814db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 monclient: found mon.noname-b 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.550+0000 7fd6f77fe640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f81f3f10 con 0x7fd6f80c7c50 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.551+0000 7fd6f77fe640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 0x7fd6f81f7330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c4d0 tx=0x7fd6e400c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.551+0000 7fd6d3fff640 10 client.?.objecter ms_handle_connect 0x7fd6f80c7c50 2026-02-22T21:42:47.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.551+0000 7fd6d3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.552+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4017070 con 0x7fd6f80c7c50 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.552+0000 7fd6d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.552+0000 7fd6d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.552+0000 7fd6d3fff640 10 monclient: dump: 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.565 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 5 monclient: authenticate success, global_id 5182 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e4005e00 con 0x7fd6f80c7c50 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 monclient: _renew_subs 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f81f41f0 con 0x7fd6f80c7c50 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e40072b0 con 0x7fd6f80c7c50 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6d3fff640 10 monclient: dump: 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.567 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6f5ffb640 4 set_mon_vals no callback set 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 client.5182.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 monclient: _renew_subs 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f806e950 con 0x7fd6f80c7c50 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 1 librados: init done 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 client.5182.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 15 client.5182.objecter _session_command_op_assign -1 1 2026-02-22T21:42:47.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 20 client.5182.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:47.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 client.5182.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:47.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.553+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd6f8069fb0 con 0x7fd6f80c7c50 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fd6e4007450 con 0x7fd6f80c7c50 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6d3fff640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd6c8078e50 0x7fd6c807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6f6ffd640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd6c8078e50 0x7fd6c807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6f6ffd640 10 monclient: get_auth_request con 0x7fd6c8078e50 auth_method 0 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6d3fff640 10 client.5182.objecter ms_handle_connect 0x7fd6c8078e50 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.565+0000 7fd6f6ffd640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd6c8078e50 0x7fd6c807b310 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8039000 tx=0x7fd6e8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fd6e4007770 con 0x7fd6f80c7c50 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 10 client.5182.objecter ms_dispatch2 0x7fd6f8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 3 client.5182.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 3 client.5182.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:47.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 10 client.5182.objecter checking command 1 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7fd6c8081330 0x7fd6c8083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter _get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 15 client.5182.objecter _session_command_op_remove -1 1 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter _get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 15 client.5182.objecter _session_command_op_assign 4 1 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 4 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 10 client.5182.objecter _send_command 1 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6e4020ec0 con 0x7fd6c8081330 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 20 client.5182.objecter dump_active .. 0 homeless 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6e405dae0 con 0x7fd6f80c7c50 2026-02-22T21:42:47.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.566+0000 7fd6d3fff640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:47.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.567+0000 7fd6f7fff640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7fd6c8081330 0x7fd6c8083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.567+0000 7fd6f7fff640 10 monclient: get_auth_request con 0x7fd6c8081330 auth_method 0 2026-02-22T21:42:47.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.568+0000 7fd6f7fff640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7fd6c8081330 0x7fd6c8083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.568+0000 7fd6d3fff640 10 client.5182.objecter ms_handle_connect 0x7fd6c8081330 2026-02-22T21:42:47.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.578+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== osd.4 v2:10.20.193.51:6810/1021106712 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fd6e4020ec0 con 0x7fd6c8081330 2026-02-22T21:42:47.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.578+0000 7fd6d3fff640 10 client.5182.objecter ms_dispatch2 0x7fd6f8000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:47.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.578+0000 7fd6d3fff640 10 client.5182.objecter _finish_command 1 = system:0 2026-02-22T21:42:47.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.578+0000 7fd6d3fff640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 4 2026-02-22T21:42:47.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.578+0000 7fd6d3fff640 15 client.5182.objecter _session_command_op_remove 4 1 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 10 client.5182.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 15 client.5182.objecter _session_command_op_assign -1 2 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 20 client.5182.objecter _get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 20 client.5182.objecter _get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 15 client.5182.objecter _session_command_op_remove -1 2 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 20 client.5182.objecter get_session s=0x7fd6c8080c60 osd=4 3 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 15 client.5182.objecter _session_command_op_assign 4 2 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 4 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 10 client.5182.objecter _send_command 2 2026-02-22T21:42:47.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.584+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 --> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd6f815af10 con 0x7fd6c8081330 2026-02-22T21:42:47.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6d3fff640 1 -- 10.20.193.51:0/1210463487 <== osd.4 v2:10.20.193.51:6810/1021106712 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7fd6f815af10 con 0x7fd6c8081330 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6d3fff640 10 client.5182.objecter ms_dispatch2 0x7fd6f8000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6d3fff640 10 client.5182.objecter _finish_command 2 = system:0 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6d3fff640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 4 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6d3fff640 15 client.5182.objecter _session_command_op_remove 4 2 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:47.600 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:47.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 10 librados: watch_flush enter 2026-02-22T21:42:47.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 10 librados: watch_flush exit 2026-02-22T21:42:47.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 10 client.5182.objecter close_session for osd.4 2026-02-22T21:42:47.602 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7fd6c8081330 msgr2=0x7fd6c8083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6810/1021106712,v1:10.20.193.51:6811/1021106712] conn(0x7fd6c8081330 0x7fd6c8083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 20 client.5182.objecter put_session s=0x7fd6c8080c60 osd=4 1 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 20 client.5182.objecter shutdown clearing up homeless session... 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 10 client.5182.objecter successfully canceled tick 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd6c8078e50 msgr2=0x7fd6c807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fd6c8078e50 0x7fd6c807b310 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8039000 tx=0x7fd6e8033000 comp rx=0 tx=0).stop 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 10 monclient: shutdown 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 msgr2=0x7fd6f81f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 --2- 10.20.193.51:0/1210463487 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fd6f80c7c50 0x7fd6f81f7330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c4d0 tx=0x7fd6e400c9a0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.603 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6f7fff640 1 -- 10.20.193.51:0/1210463487 reap_dead start 2026-02-22T21:42:47.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 shutdown_connections 2026-02-22T21:42:47.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 >> 10.20.193.51:0/1210463487 conn(0x7fd6f80c9d50 msgr2=0x7fd6f8151080 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 shutdown_connections 2026-02-22T21:42:47.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 -- 10.20.193.51:0/1210463487 wait complete. 2026-02-22T21:42:47.604 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.598+0000 7fd6fdb1b640 1 librados: shutdown 2026-02-22T21:42:47.608 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2026-02-22T21:42:47.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 10 monclient: get_monmap_and_config 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 10 monclient: monmap: 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:47.675873+0000 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:47.675873+0000 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:47.677 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:47.678 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:47.678 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:47.678 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:47.678 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:47.678 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 1 Processor -- start 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 1 -- start start 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.675+0000 7f72784f1640 10 monclient: init 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient: _add_conns ranks=[0,3,5,2,4,1] 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7270150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f727015c710 0x7f72700c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-a con 0x7f727015c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7270150600 0x7f727006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7270150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:47.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f727015a920 con 0x7f7270150220 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f727015af60 con 0x7f727015c710 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f727015b240 con 0x7f7270150600 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f72784f1640 10 monclient(hunting): authenticate will time out at 1443.256270s 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7270150600 0x7f727006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7275a65640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f727015c710 0x7f72700c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f7270150600 0x7f727006fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36704/0 (socket says 10.20.193.51:36704) 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 1 -- 10.20.193.51:0/1640448755 learned_addr learned my addr 10.20.193.51:0/1640448755 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 10 monclient(hunting): get_auth_request con 0x7f7270150600 auth_method 0 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.676+0000 7f7276a67640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): get_auth_request con 0x7f727015c710 auth_method 0 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient(hunting): handle_auth_done global_id 5162 payload 274 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 1 -- 10.20.193.51:0/1640448755 >> v2:10.20.193.51:3300/0 conn(0x7f727015c710 msgr2=0x7f72700c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 1 --2- 10.20.193.51:0/1640448755 >> v2:10.20.193.51:3300/0 conn(0x7f727015c710 0x7f72700c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 1 -- 10.20.193.51:0/1640448755 >> v1:10.20.193.51:6789/0 conn(0x7f7270150220 legacy=0x7f727015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.677+0000 7f7276a67640 1 -- 10.20.193.51:0/1640448755 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727015b520 con 0x7f7270150600 2026-02-22T21:42:47.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.678+0000 7f7276a67640 1 --2- 10.20.193.51:0/1640448755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006fca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f726c009e50 tx=0x7f726c02f330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a9f8a93d60a34ee server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 1 -- 10.20.193.51:0/1640448755 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c02fdf0 con 0x7f7270150600 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: dump: 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.691 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 5 monclient: authenticate success, global_id 5162 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 1 -- 10.20.193.51:0/1640448755 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726c0373f0 con 0x7f7270150600 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:47.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 1 -- 10.20.193.51:0/1640448755 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c0377c0 con 0x7f7270150600 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 4 set_mon_vals no callback set 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f7275264640 10 monclient: dump: 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.693 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 10 monclient: shutdown 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 -- 10.20.193.51:0/1640448755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 msgr2=0x7f727006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 --2- 10.20.193.51:0/1640448755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006fca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f726c009e50 tx=0x7f726c02f330 comp rx=0 tx=0).stop 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 -- 10.20.193.51:0/1640448755 shutdown_connections 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 --2- 10.20.193.51:0/1640448755 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006fca0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 --2- 10.20.193.51:0/1640448755 >> v2:10.20.193.51:3300/0 conn(0x7f727015c710 0x7f72700c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 -- 10.20.193.51:0/1640448755 >> 10.20.193.51:0/1640448755 conn(0x7f72700cd7b0 msgr2=0x7f72700cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 -- 10.20.193.51:0/1640448755 shutdown_connections 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.680+0000 7f72784f1640 1 -- 10.20.193.51:0/1640448755 wait complete. 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 10 monclient: monmap: 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:47.695 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 librados: starting msgr at 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 librados: starting objecter 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 Processor -- start 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 -- start start 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 librados: setting wanted keys 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 1 librados: calling monclient init 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.681+0000 7f72784f1640 10 monclient: init 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7270150220 0x7f72701f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7270150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:47.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7270150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 0x7f727014c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): picked mon.noname-a con 0x7f727015c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f72700701e0 con 0x7f727015c710 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f727015bae0 con 0x7f7270150220 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f7270070360 con 0x7f7270150600 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f72784f1640 10 monclient(hunting): authenticate will time out at 1443.262114s 2026-02-22T21:42:47.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 0x7f727014c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 0x7f727014c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35320/0 (socket says 10.20.193.51:35320) 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 1 -- 10.20.193.51:0/1191865203 learned_addr learned my addr 10.20.193.51:0/1191865203 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): get_auth_request con 0x7f727015c710 auth_method 0 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): get_auth_request con 0x7f7270150600 auth_method 0 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276266640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7270150220 0x7f72701f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276266640 10 monclient(hunting): get_auth_request con 0x7f7270150220 auth_method 0 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276266640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276266640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7275a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.682+0000 7f7276a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.683+0000 7f7276266640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.683+0000 7f7276266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.683+0000 7f7276266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 10 monclient(hunting): handle_auth_done global_id 5165 payload 824 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 -- 10.20.193.51:0/1191865203 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7270150220 msgr2=0x7f72701f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7270150220 0x7f72701f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 -- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 msgr2=0x7f727014c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 0x7f727014c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 -- 10.20.193.51:0/1191865203 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727006e0c0 con 0x7f7270150600 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7275a65640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006db80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f726400e870 tx=0x7f726400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7256ffd640 10 client.?.objecter ms_handle_connect 0x7f7270150600 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.684+0000 7f7256ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264017070 con 0x7f7270150600 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: dump: 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.701 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726400c940 con 0x7f7270150600 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 5 monclient: authenticate success, global_id 5165 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726400cd10 con 0x7f7270150600 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7256ffd640 10 monclient: dump: 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.703 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f7274a63640 4 set_mon_vals no callback set 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 10 monclient: _renew_subs 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72701e53c0 con 0x7f7270150600 2026-02-22T21:42:47.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 10 client.5165.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 10 monclient: _renew_subs 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72701e6bf0 con 0x7f7270150600 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.686+0000 7f72784f1640 1 librados: init done 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.687+0000 7f72784f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.687+0000 7f72784f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.687+0000 7f72784f1640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.687+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727006fca0 con 0x7f7270150600 2026-02-22T21:42:47.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.698+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f7264024060 con 0x7f7270150600 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.698+0000 7f7256ffd640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 0x7f724007b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.698+0000 7f7276266640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 0x7f724007b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.698+0000 7f7276266640 10 monclient: get_auth_request con 0x7f7240078e30 auth_method 0 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 10 client.5165.objecter ms_handle_connect 0x7f7240078e30 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.698+0000 7f7276266640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 0x7f724007b2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f72600051f0 tx=0x7f7260007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f726405dfb0 con 0x7f7270150600 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 10 client.5165.objecter ms_dispatch2 0x7f7270000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 3 client.5165.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 3 client.5165.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:47.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.699+0000 7f7256ffd640 20 client.5165.objecter dump_active .. 0 homeless 2026-02-22T21:42:47.805 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.804+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f7264061f70 con 0x7f7270150600 2026-02-22T21:42:47.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.804+0000 7f7256ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.806 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.804+0000 7f7256ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:47.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.886+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f7270074da0 con 0x7f7240078e30 2026-02-22T21:42:47.888 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.886+0000 7f7256ffd640 1 -- 10.20.193.51:0/1191865203 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7270074da0 con 0x7f7240078e30 2026-02-22T21:42:47.888 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 4 to deep-scrub 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 10 librados: watch_flush enter 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 10 librados: watch_flush exit 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 20 client.5165.objecter shutdown clearing up homeless session... 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 10 client.5165.objecter successfully canceled tick 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 msgr2=0x7f724007b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.890 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 0x7f724007b2f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f72600051f0 tx=0x7f7260007f30 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 10 monclient: shutdown 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 msgr2=0x7f727006db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006db80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f726400e870 tx=0x7f726400ed40 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 shutdown_connections 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7240078e30 0x7f724007b2f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f727015c710 0x7f727014c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f7270150600 0x7f727006db80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 --2- 10.20.193.51:0/1191865203 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f7270150220 0x7f72701f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 >> 10.20.193.51:0/1191865203 conn(0x7f72700cd7b0 msgr2=0x7f72700cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 shutdown_connections 2026-02-22T21:42:47.891 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 -- 10.20.193.51:0/1191865203 wait complete. 2026-02-22T21:42:47.892 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.887+0000 7f72784f1640 1 librados: shutdown 2026-02-22T21:42:47.897 INFO:tasks.ceph:Scrubbing osd.8 2026-02-22T21:42:47.897 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:47.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.964+0000 7f1fb5461640 10 monclient: get_monmap_and_config 2026-02-22T21:42:47.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.964+0000 7f1fb5461640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.964+0000 7f1fb5461640 10 monclient: monmap: 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:47.965159+0000 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:47.965159+0000 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:47.967 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.964+0000 7f1fb5461640 1 Processor -- start 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 -- start start 2026-02-22T21:42:47.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient: init 2026-02-22T21:42:47.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient: _add_conns ranks=[5,4,1,3,2,0] 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f1fb0152910 0x7f1fb00c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb0152910 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 0x7f1fb0070810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb0152cf0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f1fb0159320 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fb00c6390 con 0x7f1fb0159320 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f1fb00737a0 con 0x7f1fb0152cf0 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f1fb01597d0 con 0x7f1fb0152910 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fb5461640 10 monclient(hunting): authenticate will time out at 1443.545544s 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.965+0000 7f1fae7fc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 0x7f1fb0070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f1fb0152910 0x7f1fb00c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 0x7f1fb0070810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:40916/0 (socket says 10.20.193.51:40916) 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 1 -- 10.20.193.51:0/2366305387 learned_addr learned my addr 10.20.193.51:0/2366305387 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): get_auth_request con 0x7f1fb0152cf0 auth_method 0 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): get_auth_request con 0x7f1fb0152910 auth_method 0 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.966+0000 7f1fae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 10 monclient(hunting): handle_auth_done global_id 5168 payload 274 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 1 -- 10.20.193.51:0/2366305387 >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 msgr2=0x7f1fb0070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 1 --2- 10.20.193.51:0/2366305387 >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 0x7f1fb0070810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 1 -- 10.20.193.51:0/2366305387 >> v1:10.20.193.68:6789/0 conn(0x7f1fb0159320 legacy=0x7f1fb00c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.967+0000 7f1faeffd640 1 -- 10.20.193.51:0/2366305387 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0073a60 con 0x7f1fb0152910 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.968+0000 7f1faeffd640 1 --2- 10.20.193.51:0/2366305387 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb00c7c50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0069a40 tx=0x7f1fa402ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f92e6912c7537ba server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 1 -- 10.20.193.51:0/2366305387 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa403c070 con 0x7f1fb0152910 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: dump: 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.981 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 5 monclient: authenticate success, global_id 5168 2026-02-22T21:42:47.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 1 -- 10.20.193.51:0/2366305387 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa402faf0 con 0x7f1fb0152910 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 1 -- 10.20.193.51:0/2366305387 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4037440 con 0x7f1fb0152910 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 4 set_mon_vals no callback set 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fadffb640 10 monclient: dump: 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.983 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 10 monclient: shutdown 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 -- 10.20.193.51:0/2366305387 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 msgr2=0x7f1fb00c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 --2- 10.20.193.51:0/2366305387 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb00c7c50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0069a40 tx=0x7f1fa402ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 -- 10.20.193.51:0/2366305387 shutdown_connections 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 --2- 10.20.193.51:0/2366305387 >> v2:10.20.193.68:3300/0 conn(0x7f1fb0152cf0 0x7f1fb0070810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 --2- 10.20.193.51:0/2366305387 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb00c7c50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 -- 10.20.193.51:0/2366305387 >> 10.20.193.51:0/2366305387 conn(0x7f1fb00c9d50 msgr2=0x7f1fb00ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 -- 10.20.193.51:0/2366305387 shutdown_connections 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.970+0000 7f1fb5461640 1 -- 10.20.193.51:0/2366305387 wait complete. 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient: build_initial_monmap 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient: monmap: 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:47.985 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 librados: starting msgr at 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 librados: starting objecter 2026-02-22T21:42:47.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 Processor -- start 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 -- start start 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 librados: setting wanted keys 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 librados: calling monclient init 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient: init 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb01f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb0152910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1fb0152cf0 0x7f1fb01f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb0152cf0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:47.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1fb0159320 0x7f1fb0157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb0159320 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0074300 con 0x7f1fb0152cf0 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0074020 con 0x7f1fb0159320 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb00745e0 con 0x7f1fb0152910 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.971+0000 7f1fb5461640 10 monclient(hunting): authenticate will time out at 1443.551837s 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb01f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1fb0159320 0x7f1fb0157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb01f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36720/0 (socket says 10.20.193.51:36720) 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 1 -- 10.20.193.51:0/3817159237 learned_addr learned my addr 10.20.193.51:0/3817159237 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): get_auth_request con 0x7f1fb0152910 auth_method 0 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): get_auth_request con 0x7f1fb0159320 auth_method 0 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.972+0000 7f1faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 10 monclient(hunting): handle_auth_done global_id 5171 payload 824 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 -- 10.20.193.51:0/3817159237 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1fb0159320 msgr2=0x7f1fb0157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f1fb0159320 0x7f1fb0157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 -- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1fb0152cf0 msgr2=0x7f1fb01f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1fb0152cf0 0x7f1fb01f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:47.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 monclient: found mon.noname-c 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.973+0000 7f1faeffd640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0149ba0 con 0x7f1fb0152910 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.974+0000 7f1faeffd640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb01f7b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009130 tx=0x7f1fa40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.974+0000 7f1f8f7fe640 10 client.?.objecter ms_handle_connect 0x7f1fb0152910 2026-02-22T21:42:47.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.974+0000 7f1f8f7fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:47.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa403c040 con 0x7f1fb0152910 2026-02-22T21:42:47.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:47.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: dump: 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.996 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa4042610 con 0x7f1fb0152910 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 5 monclient: authenticate success, global_id 5171 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 10 monclient: _renew_subs 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01f4430 con 0x7f1fb0152910 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:47.997 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40429e0 con 0x7f1fb0152910 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1f8f7fe640 10 monclient: dump: 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:47.998 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fad7fa640 4 set_mon_vals no callback set 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 10 client.5171.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 10 monclient: _renew_subs 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb01f49f0 con 0x7f1fb0152910 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.976+0000 7f1fb5461640 1 librados: init done 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 10 client.5171.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 15 client.5171.objecter _session_command_op_assign -1 1 2026-02-22T21:42:47.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 20 client.5171.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 10 client.5171.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.977+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fb014a190 con 0x7f1fb0152910 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.988+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f1fa404b450 con 0x7f1fb0152910 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.988+0000 7f1f8f7fe640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1f7c078e00 0x7f1f7c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.988+0000 7f1fae7fc640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1f7c078e00 0x7f1f7c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.988+0000 7f1fae7fc640 10 monclient: get_auth_request con 0x7f1f7c078e00 auth_method 0 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.989+0000 7f1f8f7fe640 10 client.5171.objecter ms_handle_connect 0x7f1f7c078e00 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.989+0000 7f1fae7fc640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1f7c078e00 0x7f1f7c07b2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1fb01500b0 tx=0x7f1f980096b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f1fa402fa70 con 0x7f1fb0152910 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 10 client.5171.objecter ms_dispatch2 0x7f1fb0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.000 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 3 client.5171.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 3 client.5171.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 10 client.5171.objecter checking command 1 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7f1f7c0812e0 0x7f1f7c083700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter _get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 15 client.5171.objecter _session_command_op_remove -1 1 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter _get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 15 client.5171.objecter _session_command_op_assign 8 1 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 4 2026-02-22T21:42:48.001 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 10 client.5171.objecter _send_command 1 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fa4039e90 con 0x7f1f7c0812e0 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1f8f7fe640 20 client.5171.objecter dump_active .. 0 homeless 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1faf7fe640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7f1f7c0812e0 0x7f1f7c083700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.990+0000 7f1faf7fe640 10 monclient: get_auth_request con 0x7f1f7c0812e0 auth_method 0 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.992+0000 7f1faf7fe640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7f1f7c0812e0 0x7f1f7c083700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.002 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:47.992+0000 7f1f8f7fe640 10 client.5171.objecter ms_handle_connect 0x7f1f7c0812e0 2026-02-22T21:42:48.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.002+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== osd.8 v2:10.20.193.51:6826/142198174 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f1fa4039e90 con 0x7f1f7c0812e0 2026-02-22T21:42:48.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.002+0000 7f1f8f7fe640 10 client.5171.objecter ms_dispatch2 0x7f1fb0000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:48.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.002+0000 7f1f8f7fe640 10 client.5171.objecter _finish_command 1 = system:0 2026-02-22T21:42:48.004 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.003+0000 7f1f8f7fe640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 4 2026-02-22T21:42:48.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.003+0000 7f1f8f7fe640 15 client.5171.objecter _session_command_op_remove 8 1 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 10 client.5171.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 15 client.5171.objecter _session_command_op_assign -1 2 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 20 client.5171.objecter _get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 20 client.5171.objecter _get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 15 client.5171.objecter _session_command_op_remove -1 2 2026-02-22T21:42:48.011 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 20 client.5171.objecter get_session s=0x7f1f7c080c10 osd=8 3 2026-02-22T21:42:48.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 15 client.5171.objecter _session_command_op_assign 8 2 2026-02-22T21:42:48.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 4 2026-02-22T21:42:48.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 10 client.5171.objecter _send_command 2 2026-02-22T21:42:48.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.009+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 --> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1fb00c6560 con 0x7f1f7c0812e0 2026-02-22T21:42:48.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1f8f7fe640 1 -- 10.20.193.51:0/3817159237 <== osd.8 v2:10.20.193.51:6826/142198174 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1fb00c6560 con 0x7f1f7c0812e0 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1f8f7fe640 10 client.5171.objecter ms_dispatch2 0x7f1fb0000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1f8f7fe640 10 client.5171.objecter _finish_command 2 = system:0 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1f8f7fe640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 4 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1f8f7fe640 15 client.5171.objecter _session_command_op_remove 8 2 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:48.018 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:48.019 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:48.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 10 librados: watch_flush enter 2026-02-22T21:42:48.020 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 10 librados: watch_flush exit 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 10 client.5171.objecter close_session for osd.8 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7f1f7c0812e0 msgr2=0x7f1f7c083700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6826/142198174,v1:10.20.193.51:6827/142198174] conn(0x7f1f7c0812e0 0x7f1f7c083700 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 20 client.5171.objecter put_session s=0x7f1f7c080c10 osd=8 1 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 20 client.5171.objecter shutdown clearing up homeless session... 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 10 client.5171.objecter successfully canceled tick 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1f7c078e00 msgr2=0x7f1f7c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1f7c078e00 0x7f1f7c07b2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1fb01500b0 tx=0x7f1f980096b0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 10 monclient: shutdown 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 20 monclient: shutdown canceling and discarding version request 1 2026-02-22T21:42:48.021 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 msgr2=0x7f1fb01f7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 --2- 10.20.193.51:0/3817159237 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f1fb0152910 0x7f1fb01f7b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009130 tx=0x7f1fa40047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1faf7fe640 1 -- 10.20.193.51:0/3817159237 reap_dead start 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 shutdown_connections 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 >> 10.20.193.51:0/3817159237 conn(0x7f1fb00c9d50 msgr2=0x7f1fb0150d20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 shutdown_connections 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 -- 10.20.193.51:0/3817159237 wait complete. 2026-02-22T21:42:48.022 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.016+0000 7f1fb5461640 1 librados: shutdown 2026-02-22T21:42:48.026 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.093+0000 7fa446edb640 10 monclient: get_monmap_and_config 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.093+0000 7fa446edb640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.093+0000 7fa446edb640 10 monclient: monmap: 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:48.094493+0000 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:48.094493+0000 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:48.095 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 Processor -- start 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 -- start start 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient: init 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient: _add_conns ranks=[3,4,2,0,1,5] 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 0x7fa4400c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-a con 0x7fa440152ab0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa44006d9d0 0x7fa4400c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-b con 0x7fa44006d9d0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4400c5ff0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44006f1d0 con 0x7fa4400c5ff0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa44006ed40 con 0x7fa440152ab0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fa44006de50 con 0x7fa44006d9d0 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.094+0000 7fa446edb640 10 monclient(hunting): authenticate will time out at 1443.674870s 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 0x7fa4400c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fa44006d9d0 0x7fa4400c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa445451640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fa4400c5ff0 0x7fa44006fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47540/0 (socket says 10.20.193.51:47540) 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 0x7fa4400c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35352/0 (socket says 10.20.193.51:35352) 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa445451640 1 -- 10.20.193.51:0/1305404431 learned_addr learned my addr 10.20.193.51:0/1305404431 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): get_auth_request con 0x7fa440152ab0 auth_method 0 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): get_auth_request con 0x7fa44006d9d0 auth_method 0 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.095+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 10 monclient(hunting): handle_auth_done global_id 5194 payload 274 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 1 -- 10.20.193.51:0/1305404431 >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 msgr2=0x7fa4400c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 1 --2- 10.20.193.51:0/1305404431 >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 0x7fa4400c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.097 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 1 -- 10.20.193.51:0/1305404431 >> v1:10.20.193.125:6789/0 conn(0x7fa4400c5ff0 legacy=0x7fa44006fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 1 monclient: found mon.noname-b 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.096+0000 7fa437fff640 1 -- 10.20.193.51:0/1305404431 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44006e130 con 0x7fa44006d9d0 2026-02-22T21:42:48.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.097+0000 7fa437fff640 1 --2- 10.20.193.51:0/1305404431 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa44006d9d0 0x7fa4400c8e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa44006eec0 tx=0x7fa42802eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=318702b5676a5f11 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.098+0000 7fa4377fe640 1 -- 10.20.193.51:0/1305404431 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42803c070 con 0x7fa44006d9d0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: dump: 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 1 -- 10.20.193.51:0/1305404431 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42802fba0 con 0x7fa44006d9d0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 5 monclient: authenticate success, global_id 5194 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 1 -- 10.20.193.51:0/1305404431 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428037440 con 0x7fa44006d9d0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 4 set_mon_vals no callback set 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa4377fe640 10 monclient: dump: 2026-02-22T21:42:48.100 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 10 monclient: shutdown 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 -- 10.20.193.51:0/1305404431 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa44006d9d0 msgr2=0x7fa4400c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 --2- 10.20.193.51:0/1305404431 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa44006d9d0 0x7fa4400c8e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa44006eec0 tx=0x7fa42802eed0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 -- 10.20.193.51:0/1305404431 shutdown_connections 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 --2- 10.20.193.51:0/1305404431 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa44006d9d0 0x7fa4400c8e90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 --2- 10.20.193.51:0/1305404431 >> v2:10.20.193.51:3300/0 conn(0x7fa440152ab0 0x7fa4400c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 -- 10.20.193.51:0/1305404431 >> 10.20.193.51:0/1305404431 conn(0x7fa4400cd7b0 msgr2=0x7fa4400cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 -- 10.20.193.51:0/1305404431 shutdown_connections 2026-02-22T21:42:48.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 1 -- 10.20.193.51:0/1305404431 wait complete. 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.099+0000 7fa446edb640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient: monmap: 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 librados: starting msgr at 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 librados: starting objecter 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 Processor -- start 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 -- start start 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 librados: setting wanted keys 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 librados: calling monclient init 2026-02-22T21:42:48.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient: init 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-a con 0x7fa44006d9d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa4400c5ff0 0x7fa4401486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-c con 0x7fa4400c5ff0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa440152ab0 0x7fa44015ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): picked mon.noname-b con 0x7fa440152ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa44006e9d0 con 0x7fa44006d9d0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa44006e6f0 con 0x7fa440152ab0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa440152e90 con 0x7fa4400c5ff0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.100+0000 7fa446edb640 10 monclient(hunting): authenticate will time out at 1443.680872s 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35366/0 (socket says 10.20.193.51:35366) 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 1 -- 10.20.193.51:0/888549407 learned_addr learned my addr 10.20.193.51:0/888549407 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): get_auth_request con 0x7fa44006d9d0 auth_method 0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa4400c5ff0 0x7fa4401486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa440152ab0 0x7fa44015ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): get_auth_request con 0x7fa4400c5ff0 auth_method 0 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.103 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): get_auth_request con 0x7fa440152ab0 auth_method 0 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa444c50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa437fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.101+0000 7fa445451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 10 monclient(hunting): handle_auth_done global_id 5193 payload 824 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 -- 10.20.193.51:0/888549407 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa4400c5ff0 msgr2=0x7fa4401486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa4400c5ff0 0x7fa4401486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 -- 10.20.193.51:0/888549407 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa440152ab0 msgr2=0x7fa44015ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa440152ab0 0x7fa44015ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 monclient: found mon.noname-a 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.102+0000 7fa444c50640 1 -- 10.20.193.51:0/888549407 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4400c1610 con 0x7fa44006d9d0 2026-02-22T21:42:48.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.103+0000 7fa444c50640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa43000e890 tx=0x7fa43000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.103+0000 7fa4357fa640 10 client.?.objecter ms_handle_connect 0x7fa44006d9d0 2026-02-22T21:42:48.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.103+0000 7fa4357fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430017070 con 0x7fa44006d9d0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: dump: 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43000c940 con 0x7fa44006d9d0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43000cd30 con 0x7fa44006d9d0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa4357fa640 10 monclient: dump: 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.107 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa436ffd640 4 set_mon_vals no callback set 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 5 monclient: authenticate success, global_id 5193 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 10 monclient: _renew_subs 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4401f3950 con 0x7fa44006d9d0 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 10 client.5193.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 10 monclient: _renew_subs 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401f3f10 con 0x7fa44006d9d0 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.105+0000 7fa446edb640 1 librados: init done 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.106+0000 7fa446edb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.106+0000 7fa446edb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.106+0000 7fa446edb640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.106+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4400c5ab0 con 0x7fa44006d9d0 2026-02-22T21:42:48.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa430013470 con 0x7fa44006d9d0 2026-02-22T21:42:48.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa4357fa640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 0x7fa41007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa437fff640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 0x7fa41007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa437fff640 10 monclient: get_auth_request con 0x7fa410078e00 auth_method 0 2026-02-22T21:42:48.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa4357fa640 10 client.5193.objecter ms_handle_connect 0x7fa410078e00 2026-02-22T21:42:48.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.118+0000 7fa437fff640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 0x7fa41007b2c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa428039000 tx=0x7fa428033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.223 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa43005e450 con 0x7fa44006d9d0 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 10 client.5193.objecter ms_dispatch2 0x7fa440000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 3 client.5193.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 3 client.5193.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 20 client.5193.objecter dump_active .. 0 homeless 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa430062410 con 0x7fa44006d9d0 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.224 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.221+0000 7fa4357fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:48.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.306+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "8", "target": ["mon-mgr", ""]}) -- 0x7fa4400c6ec0 con 0x7fa410078e00 2026-02-22T21:42:48.307 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.306+0000 7fa4357fa640 1 -- 10.20.193.51:0/888549407 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 8 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa4400c6ec0 con 0x7fa410078e00 2026-02-22T21:42:48.307 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 8 to deep-scrub 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.306+0000 7fa446edb640 10 librados: watch_flush enter 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.306+0000 7fa446edb640 10 librados: watch_flush exit 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 20 client.5193.objecter shutdown clearing up homeless session... 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 10 client.5193.objecter successfully canceled tick 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 msgr2=0x7fa41007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 0x7fa41007b2c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa428039000 tx=0x7fa428033000 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 10 monclient: shutdown 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 msgr2=0x7fa4400c10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa43000e890 tx=0x7fa43000ed60 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 shutdown_connections 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa410078e00 0x7fa41007b2c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa440152ab0 0x7fa44015ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa4400c5ff0 0x7fa4401486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 --2- 10.20.193.51:0/888549407 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa44006d9d0 0x7fa4400c10d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 >> 10.20.193.51:0/888549407 conn(0x7fa4400cd7b0 msgr2=0x7fa440150540 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 shutdown_connections 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 -- 10.20.193.51:0/888549407 wait complete. 2026-02-22T21:42:48.308 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.307+0000 7fa446edb640 1 librados: shutdown 2026-02-22T21:42:48.316 INFO:tasks.ceph:Scrubbing osd.12 2026-02-22T21:42:48.316 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:48.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 10 monclient: get_monmap_and_config 2026-02-22T21:42:48.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 10 monclient: monmap: 2026-02-22T21:42:48.384 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:48.383123+0000 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:48.383123+0000 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:48.385 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:48.386 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 1 Processor -- start 2026-02-22T21:42:48.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 1 -- start start 2026-02-22T21:42:48.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.382+0000 7f4a22a6d640 10 monclient: init 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient: _add_conns ranks=[2,3,5,4,0,1] 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a1c152910 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4a1c152cf0 0x7f4a1c0c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a1c152cf0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4a1c070810 0x7f4a1c159320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a1c070810 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:48.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1c070cc0 con 0x7f4a1c152910 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f4a1c159860 con 0x7f4a1c152cf0 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f4a1c0c6390 con 0x7f4a1c070810 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.383+0000 7f4a22a6d640 10 monclient(hunting): authenticate will time out at 1443.963542s 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f4a1c152cf0 0x7f4a1c0c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1bfff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f4a1c152910 0x7f4a1c0c5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47556/0 (socket says 10.20.193.51:47556) 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1bfff640 1 -- 10.20.193.51:0/193037272 learned_addr learned my addr 10.20.193.51:0/193037272 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): get_auth_request con 0x7f4a1c152cf0 auth_method 0 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.384+0000 7f4a20fe3640 1 --2- 10.20.193.51:0/193037272 >> v2:10.20.193.125:3300/0 conn(0x7f4a1c070810 0x7f4a1c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a20fe3640 10 monclient(hunting): get_auth_request con 0x7f4a1c070810 auth_method 0 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a20fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a20fe3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a20fe3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_done global_id 5196 payload 274 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 1 -- 10.20.193.51:0/193037272 >> v2:10.20.193.125:3300/0 conn(0x7f4a1c070810 msgr2=0x7f4a1c159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 1 --2- 10.20.193.51:0/193037272 >> v2:10.20.193.125:3300/0 conn(0x7f4a1c070810 0x7f4a1c159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 1 -- 10.20.193.51:0/193037272 >> v1:10.20.193.125:6789/0 conn(0x7f4a1c152910 legacy=0x7f4a1c0c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.385+0000 7f4a1b7fe640 1 -- 10.20.193.51:0/193037272 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c0737a0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.386+0000 7f4a1b7fe640 1 --2- 10.20.193.51:0/193037272 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c0c7c50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0c9bf0 tx=0x7f4a0c02ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08504f958028300 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 1 -- 10.20.193.51:0/193037272 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c03c070 con 0x7f4a1c152cf0 2026-02-22T21:42:48.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: dump: 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.400 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 1 -- 10.20.193.51:0/193037272 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a0c02faf0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 5 monclient: authenticate success, global_id 5196 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 1 -- 10.20.193.51:0/193037272 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c037440 con 0x7f4a1c152cf0 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 4 set_mon_vals no callback set 2026-02-22T21:42:48.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a1affd640 10 monclient: dump: 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.402 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 10 monclient: shutdown 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 -- 10.20.193.51:0/193037272 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 msgr2=0x7f4a1c0c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/193037272 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c0c7c50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0c9bf0 tx=0x7f4a0c02ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 -- 10.20.193.51:0/193037272 shutdown_connections 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/193037272 >> v2:10.20.193.125:3300/0 conn(0x7f4a1c070810 0x7f4a1c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/193037272 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c0c7c50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 -- 10.20.193.51:0/193037272 >> 10.20.193.51:0/193037272 conn(0x7f4a1c0c9d50 msgr2=0x7f4a1c0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 -- 10.20.193.51:0/193037272 shutdown_connections 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.388+0000 7f4a22a6d640 1 -- 10.20.193.51:0/193037272 wait complete. 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 10 monclient: monmap: 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 librados: starting msgr at 2026-02-22T21:42:48.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 librados: starting objecter 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 Processor -- start 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 -- start start 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 librados: setting wanted keys 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 1 librados: calling monclient init 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.389+0000 7f4a22a6d640 10 monclient: init 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4a1c070810 0x7f4a1c147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a1c070810 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4a1c152910 0x7f4a1c14ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a1c152910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c14aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a1c152cf0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c074020 con 0x7f4a1c152cf0 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c073d40 con 0x7f4a1c152910 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c074300 con 0x7f4a1c070810 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a22a6d640 10 monclient(hunting): authenticate will time out at 1443.970221s 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c14aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c14aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35382/0 (socket says 10.20.193.51:35382) 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 1 -- 10.20.193.51:0/3374938780 learned_addr learned my addr 10.20.193.51:0/3374938780 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): get_auth_request con 0x7f4a1c152cf0 auth_method 0 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1b7fe640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4a1c152910 0x7f4a1c14ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1bfff640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4a1c070810 0x7f4a1c147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a20fe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1bfff640 10 monclient(hunting): get_auth_request con 0x7f4a1c070810 auth_method 0 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1b7fe640 10 monclient(hunting): get_auth_request con 0x7f4a1c152910 auth_method 0 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.390+0000 7f4a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.391+0000 7f4a1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 10 monclient(hunting): handle_auth_done global_id 5199 payload 824 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 -- 10.20.193.51:0/3374938780 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4a1c070810 msgr2=0x7f4a1c147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4a1c070810 0x7f4a1c147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 -- 10.20.193.51:0/3374938780 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4a1c152910 msgr2=0x7f4a1c14ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4a1c152910 0x7f4a1c14ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 monclient: found mon.noname-a 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c1f59a0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a20fe3640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c14aea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a10007d90 tx=0x7f4a1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a18ff9640 10 client.?.objecter ms_handle_connect 0x7f4a1c152cf0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.392+0000 7f4a18ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10014420 con 0x7f4a1c152cf0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: dump: 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.410 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a10019070 con 0x7f4a1c152cf0 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10014800 con 0x7f4a1c152cf0 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a18ff9640 10 monclient: dump: 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.412 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a1a7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 5 monclient: authenticate success, global_id 5199 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 10 monclient: _renew_subs 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1c1f19c0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 10 client.5199.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 10 monclient: _renew_subs 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1c1f1f80 con 0x7f4a1c152cf0 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.394+0000 7f4a22a6d640 1 librados: init done 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 10 client.5199.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 15 client.5199.objecter _session_command_op_assign -1 1 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 20 client.5199.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 10 client.5199.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.395+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a1c1f2760 con 0x7f4a1c152cf0 2026-02-22T21:42:48.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4a100149a0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a18ff9640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f49ec078e30 0x7f49ec07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a1bfff640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f49ec078e30 0x7f49ec07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a1bfff640 10 monclient: get_auth_request con 0x7f49ec078e30 auth_method 0 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a18ff9640 10 client.5199.objecter ms_handle_connect 0x7f49ec078e30 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.407+0000 7f4a1bfff640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f49ec078e30 0x7f49ec07b2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4a08002770 tx=0x7f4a08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f4a100987f0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 10 client.5199.objecter ms_dispatch2 0x7f4a1c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 3 client.5199.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 3 client.5199.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 10 client.5199.objecter checking command 1 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f49ec081370 0x7f49ec083790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter _get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 15 client.5199.objecter _session_command_op_remove -1 1 2026-02-22T21:42:48.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter _get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 15 client.5199.objecter _session_command_op_assign 12 1 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 4 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 10 client.5199.objecter _send_command 1 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a100048f0 con 0x7f49ec081370 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 20 client.5199.objecter dump_active .. 0 homeless 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a1005f2b0 con 0x7f4a1c152cf0 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.409+0000 7f4a18ff9640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.410+0000 7f4a1b7fe640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f49ec081370 0x7f49ec083790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.410+0000 7f4a1b7fe640 10 monclient: get_auth_request con 0x7f49ec081370 auth_method 0 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.412+0000 7f4a1b7fe640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f49ec081370 0x7f49ec083790 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.12 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.412+0000 7f4a18ff9640 10 client.5199.objecter ms_handle_connect 0x7f49ec081370 2026-02-22T21:42:48.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.421+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== osd.12 v2:10.20.193.51:6818/1676524147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f4a100048f0 con 0x7f49ec081370 2026-02-22T21:42:48.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.421+0000 7f4a18ff9640 10 client.5199.objecter ms_dispatch2 0x7f4a1c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:48.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.421+0000 7f4a18ff9640 10 client.5199.objecter _finish_command 1 = system:0 2026-02-22T21:42:48.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.421+0000 7f4a18ff9640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 4 2026-02-22T21:42:48.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.421+0000 7f4a18ff9640 15 client.5199.objecter _session_command_op_remove 12 1 2026-02-22T21:42:48.429 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 10 client.5199.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 15 client.5199.objecter _session_command_op_assign -1 2 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 20 client.5199.objecter _get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 20 client.5199.objecter _get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 15 client.5199.objecter _session_command_op_remove -1 2 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 20 client.5199.objecter get_session s=0x7f49ec080ca0 osd=12 3 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 15 client.5199.objecter _session_command_op_assign 12 2 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 4 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 10 client.5199.objecter _send_command 2 2026-02-22T21:42:48.430 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.427+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 --> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4a1c0745e0 con 0x7f49ec081370 2026-02-22T21:42:48.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a18ff9640 1 -- 10.20.193.51:0/3374938780 <== osd.12 v2:10.20.193.51:6818/1676524147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f4a1c0745e0 con 0x7f49ec081370 2026-02-22T21:42:48.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a18ff9640 10 client.5199.objecter ms_dispatch2 0x7f4a1c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:48.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a18ff9640 10 client.5199.objecter _finish_command 2 = system:0 2026-02-22T21:42:48.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a18ff9640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 4 2026-02-22T21:42:48.443 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a18ff9640 15 client.5199.objecter _session_command_op_remove 12 2 2026-02-22T21:42:48.444 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:48.444 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:48.444 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:48.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 10 librados: watch_flush enter 2026-02-22T21:42:48.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 10 librados: watch_flush exit 2026-02-22T21:42:48.445 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 10 client.5199.objecter close_session for osd.12 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f49ec081370 msgr2=0x7f49ec083790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6818/1676524147,v1:10.20.193.51:6819/1676524147] conn(0x7f49ec081370 0x7f49ec083790 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 20 client.5199.objecter put_session s=0x7f49ec080ca0 osd=12 1 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 20 client.5199.objecter shutdown clearing up homeless session... 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 10 client.5199.objecter successfully canceled tick 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f49ec078e30 msgr2=0x7f49ec07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f49ec078e30 0x7f49ec07b2f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4a08002770 tx=0x7f4a08009040 comp rx=0 tx=0).stop 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 10 monclient: shutdown 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 msgr2=0x7f4a1c14aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.446 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 --2- 10.20.193.51:0/3374938780 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f4a1c152cf0 0x7f4a1c14aea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a10007d90 tx=0x7f4a1000e4b0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a20fe3640 1 -- 10.20.193.51:0/3374938780 reap_dead start 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 shutdown_connections 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 >> 10.20.193.51:0/3374938780 conn(0x7f4a1c0c9d50 msgr2=0x7f4a1c0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 shutdown_connections 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 -- 10.20.193.51:0/3374938780 wait complete. 2026-02-22T21:42:48.447 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.441+0000 7f4a22a6d640 1 librados: shutdown 2026-02-22T21:42:48.451 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.516+0000 7fdd8f1d6640 10 monclient: get_monmap_and_config 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.516+0000 7fdd8f1d6640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.516+0000 7fdd8f1d6640 10 monclient: monmap: 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:48.517506+0000 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:48.517506+0000 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:48.518 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 1 Processor -- start 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 1 -- start start 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient: init 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient: _add_conns ranks=[0,1,2,4,5,3] 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd88150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fdd8815c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd880703b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8815a920 con 0x7fdd88150220 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8815af60 con 0x7fdd8815c710 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8815b240 con 0x7fdd880703b0 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.517+0000 7fdd8f1d6640 10 monclient(hunting): authenticate will time out at 1444.097876s 2026-02-22T21:42:48.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.518+0000 7fdd8cf4b640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fdd88150220 0x7fdd8815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58954/0 (socket says 10.20.193.51:58954) 2026-02-22T21:42:48.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.518+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3188652052 learned_addr learned my addr 10.20.193.51:0/3188652052 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108576314 0 0) 0x7fdd8815b240 con 0x7fdd880703b0 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): my global_id is 5189 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd74003620 con 0x7fdd880703b0 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347810209 0 0) 0x7fdd8815af60 con 0x7fdd8815c710 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): my global_id is 5203 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd8815b240 con 0x7fdd8815c710 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726178988 0 0) 0x7fdd8815a920 con 0x7fdd88150220 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _negotiate 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 10 monclient(hunting): my global_id is 5202 2026-02-22T21:42:48.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.519+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd8815af60 con 0x7fdd88150220 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2150317756 0 0) 0x7fdd8815b240 con 0x7fdd8815c710 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 >> v1:10.20.193.125:6789/0 conn(0x7fdd880703b0 legacy=0x7fdd88159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 >> v1:10.20.193.51:6789/0 conn(0x7fdd88150220 legacy=0x7fdd8815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8815b520 con 0x7fdd8815c710 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7fdd70002ee0 con 0x7fdd8815c710 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: dump: 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 >> v1:10.20.193.68:6789/0 conn(0x7fdd8815c710 legacy=0x7fdd8806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 0x7fdd74004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): picked mon.c con 0x7fdd74003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74004e40 0x7fdd740052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): picked mon.b con 0x7fdd74004e40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd74009ab0 0x7fdd7400bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): picked mon.a con 0x7fdd74009ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.523 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7400c490 con 0x7fdd74009ab0 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7400c640 con 0x7fdd74004e40 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7400c840 con 0x7fdd74003fe0 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd7f7fe640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.521+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 0x7fdd74004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): get_auth_request con 0x7fdd74003fe0 auth_method 0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd74009ab0 0x7fdd7400bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): get_auth_request con 0x7fdd74009ab0 auth_method 0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd7ffff640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74004e40 0x7fdd740052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd7ffff640 10 monclient(hunting): get_auth_request con 0x7fdd74004e40 auth_method 0 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd7ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.522+0000 7fdd7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_done global_id 5203 payload 293 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3188652052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74004e40 msgr2=0x7fdd740052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74004e40 0x7fdd740052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3188652052 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd74009ab0 msgr2=0x7fdd7400bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd74009ab0 0x7fdd7400bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 monclient: found mon.c 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3188652052 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8815b240 con 0x7fdd74003fe0 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8d74c640 1 -- 10.20.193.51:0/3188652052 reap_dead start 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8cf4b640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8f1d6640 5 monclient: authenticate success, global_id 5203 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.523+0000 7fdd8f1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:48.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.524+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 0x7fdd74004900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd78003280 tx=0x7fdd7802efe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dae6fae8792cf1bc server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78002ea0 con 0x7fdd74003fe0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: dump: 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd78033070 con 0x7fdd74003fe0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 1 -- 10.20.193.51:0/3188652052 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78004de0 con 0x7fdd74003fe0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd7f7fe640 10 monclient: dump: 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.527 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.528 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.528 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.528 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:48.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 4 set_mon_vals no callback set 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 10 monclient: shutdown 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 msgr2=0x7fdd74004900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.525+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 0x7fdd74004900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd78003280 tx=0x7fdd7802efe0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3188652052 shutdown_connections 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3188652052 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd74003fe0 0x7fdd74004900 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3188652052 >> 10.20.193.51:0/3188652052 conn(0x7fdd880cd7b0 msgr2=0x7fdd880cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3188652052 shutdown_connections 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3188652052 wait complete. 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 10 monclient: monmap: 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 librados: starting msgr at 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 librados: starting objecter 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 Processor -- start 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 -- start start 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 librados: setting wanted keys 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 1 librados: calling monclient init 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.526+0000 7fdd8f1d6640 10 monclient: init 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 0x7fdd880c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd74003fe0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 0x7fdd881f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd880bf4d0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd881f4500 0x7fdd8814c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd881f4500 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88150600 con 0x7fdd880bf4d0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8815bae0 con 0x7fdd74003fe0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88150780 con 0x7fdd881f4500 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8f1d6640 10 monclient(hunting): authenticate will time out at 1444.107240s 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 0x7fdd881f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8cf4b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 0x7fdd880c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8d74c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd881f4500 0x7fdd8814c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 0x7fdd881f7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35392/0 (socket says 10.20.193.51:35392) 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 1 -- 10.20.193.51:0/3833760855 learned_addr learned my addr 10.20.193.51:0/3833760855 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 10 monclient(hunting): get_auth_request con 0x7fdd880bf4d0 auth_method 0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd7ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8cf4b640 10 monclient(hunting): get_auth_request con 0x7fdd74003fe0 auth_method 0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8cf4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8cf4b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8cf4b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8d74c640 10 monclient(hunting): get_auth_request con 0x7fdd881f4500 auth_method 0 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8d74c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8d74c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.527+0000 7fdd8d74c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8d74c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.528+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 10 monclient(hunting): handle_auth_done global_id 5206 payload 824 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3833760855 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd881f4500 msgr2=0x7fdd8814c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd881f4500 0x7fdd8814c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 msgr2=0x7fdd881f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 0x7fdd881f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 monclient: found mon.noname-b 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.529+0000 7fdd8cf4b640 1 -- 10.20.193.51:0/3833760855 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd881f7750 con 0x7fdd74003fe0 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.530+0000 7fdd8cf4b640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 0x7fdd880c80e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400ca10 tx=0x7fdd78002ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.530+0000 7fdd7d7fa640 10 client.?.objecter ms_handle_connect 0x7fdd74003fe0 2026-02-22T21:42:48.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.530+0000 7fdd7d7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7802fa10 con 0x7fdd74003fe0 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: dump: 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.533 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 5 monclient: authenticate success, global_id 5206 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 10 monclient: _renew_subs 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd881f7a30 con 0x7fdd74003fe0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 10 client.5206.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 10 monclient: _renew_subs 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd881f3d00 con 0x7fdd74003fe0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd8f1d6640 1 librados: init done 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd78033060 con 0x7fdd74003fe0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7effd640 4 set_mon_vals no callback set 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7804b060 con 0x7fdd74003fe0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.531+0000 7fdd7d7fa640 10 monclient: dump: 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.534 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.532+0000 7fdd8f1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.532+0000 7fdd8f1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.532+0000 7fdd8f1d6640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.532+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd88074da0 con 0x7fdd74003fe0 2026-02-22T21:42:48.638 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.637+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fdd78007070 con 0x7fdd74003fe0 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.637+0000 7fdd7d7fa640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 0x7fdd5807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.637+0000 7fdd7ffff640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 0x7fdd5807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7ffff640 10 monclient: get_auth_request con 0x7fdd58078e30 auth_method 0 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7ffff640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 0x7fdd5807b2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd881e91a0 tx=0x7fdd70009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 10 client.5206.objecter ms_handle_connect 0x7fdd58078e30 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fdd7802fdb0 con 0x7fdd74003fe0 2026-02-22T21:42:48.640 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 10 client.5206.objecter ms_dispatch2 0x7fdd88000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 3 client.5206.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 3 client.5206.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.641 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.638+0000 7fdd7d7fa640 20 client.5206.objecter dump_active .. 0 homeless 2026-02-22T21:42:48.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.645+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fdd78085ce0 con 0x7fdd74003fe0 2026-02-22T21:42:48.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.645+0000 7fdd7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.646 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.645+0000 7fdd7d7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:48.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.726+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "12", "target": ["mon-mgr", ""]}) -- 0x7fdd88158fe0 con 0x7fdd58078e30 2026-02-22T21:42:48.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.726+0000 7fdd7d7fa640 1 -- 10.20.193.51:0/3833760855 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 12 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7fdd88158fe0 con 0x7fdd58078e30 2026-02-22T21:42:48.728 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 12 to deep-scrub 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 10 librados: watch_flush enter 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 10 librados: watch_flush exit 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 20 client.5206.objecter shutdown clearing up homeless session... 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 10 client.5206.objecter successfully canceled tick 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 msgr2=0x7fdd5807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 0x7fdd5807b2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd881e91a0 tx=0x7fdd70009040 comp rx=0 tx=0).stop 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 10 monclient: shutdown 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 msgr2=0x7fdd880c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 0x7fdd880c80e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400ca10 tx=0x7fdd78002ac0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 shutdown_connections 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fdd58078e30 0x7fdd5807b2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fdd881f4500 0x7fdd8814c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fdd880bf4d0 0x7fdd881f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 --2- 10.20.193.51:0/3833760855 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fdd74003fe0 0x7fdd880c80e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 >> 10.20.193.51:0/3833760855 conn(0x7fdd880cd7b0 msgr2=0x7fdd880c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 shutdown_connections 2026-02-22T21:42:48.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 -- 10.20.193.51:0/3833760855 wait complete. 2026-02-22T21:42:48.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.727+0000 7fdd8f1d6640 1 librados: shutdown 2026-02-22T21:42:48.738 INFO:tasks.ceph:Scrubbing osd.1 2026-02-22T21:42:48.738 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 10 monclient: get_monmap_and_config 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 10 monclient: monmap: 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.809 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:48.807865+0000 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:48.807865+0000 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:48.810 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 1 Processor -- start 2026-02-22T21:42:48.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 1 -- start start 2026-02-22T21:42:48.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.807+0000 7f88aba6d640 10 monclient: init 2026-02-22T21:42:48.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient: _add_conns ranks=[2,5,3,0,4,1] 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88a4152910 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f88a4152cf0 0x7f88a40c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-c con 0x7f88a4152cf0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f88a4070810 0x7f88a4159320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-a con 0x7f88a4070810 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a4070cc0 con 0x7f88a4152910 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f88a4159860 con 0x7f88a4070810 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f88a40c6390 con 0x7f88a4152cf0 2026-02-22T21:42:48.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88aba6d640 10 monclient(hunting): authenticate will time out at 1444.388287s 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f88a4070810 0x7f88a4159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f88a4070810 0x7f88a4159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35408/0 (socket says 10.20.193.51:35408) 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 1 -- 10.20.193.51:0/47446575 learned_addr learned my addr 10.20.193.51:0/47446575 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 10 monclient(hunting): get_auth_request con 0x7f88a4070810 auth_method 0 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a8fe1640 1 --2- 10.20.193.51:0/47446575 >> v2:10.20.193.125:3300/0 conn(0x7f88a4152cf0 0x7f88a40c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.808+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): get_auth_request con 0x7f88a4152cf0 auth_method 0 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_done global_id 5208 payload 274 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 1 -- 10.20.193.51:0/47446575 >> v2:10.20.193.125:3300/0 conn(0x7f88a4152cf0 msgr2=0x7f88a40c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 1 --2- 10.20.193.51:0/47446575 >> v2:10.20.193.125:3300/0 conn(0x7f88a4152cf0 0x7f88a40c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 1 -- 10.20.193.51:0/47446575 >> v1:10.20.193.125:6789/0 conn(0x7f88a4152910 legacy=0x7f88a40c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 1 monclient: found mon.noname-a 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.809+0000 7f88a9fe3640 1 -- 10.20.193.51:0/47446575 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a40737a0 con 0x7f88a4070810 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.810+0000 7f88a9fe3640 1 --2- 10.20.193.51:0/47446575 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4070810 0x7f88a4159320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f88a406cd60 tx=0x7f88a002f270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af115a62fc478a62 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 1 -- 10.20.193.51:0/47446575 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a002fd70 con 0x7f88a4070810 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 10 monclient: dump: 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.822 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f8893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f88aba6d640 5 monclient: authenticate success, global_id 5208 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.811+0000 7f88aba6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 1 -- 10.20.193.51:0/47446575 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a00373f0 con 0x7f88a4070810 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 1 -- 10.20.193.51:0/47446575 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a00377c0 con 0x7f88a4070810 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f8893fff640 10 monclient: dump: 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.824 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 4 set_mon_vals no callback set 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 10 monclient: shutdown 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 1 -- 10.20.193.51:0/47446575 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4070810 msgr2=0x7f88a4159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.812+0000 7f88aba6d640 1 --2- 10.20.193.51:0/47446575 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4070810 0x7f88a4159320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f88a406cd60 tx=0x7f88a002f270 comp rx=0 tx=0).stop 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 -- 10.20.193.51:0/47446575 shutdown_connections 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 --2- 10.20.193.51:0/47446575 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4070810 0x7f88a4159320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 --2- 10.20.193.51:0/47446575 >> v2:10.20.193.125:3300/0 conn(0x7f88a4152cf0 0x7f88a40c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 -- 10.20.193.51:0/47446575 >> 10.20.193.51:0/47446575 conn(0x7f88a40c9d50 msgr2=0x7f88a40ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 -- 10.20.193.51:0/47446575 shutdown_connections 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 -- 10.20.193.51:0/47446575 wait complete. 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 10 monclient: monmap: 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:48.826 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 librados: starting msgr at 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 librados: starting objecter 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 Processor -- start 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 -- start start 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 librados: setting wanted keys 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 1 librados: calling monclient init 2026-02-22T21:42:48.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.813+0000 7f88aba6d640 10 monclient: init 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88a4070810 0x7f88a41f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-c con 0x7f88a4070810 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88a4152910 0x7f88a4165850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-b con 0x7f88a4152910 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 0x7f88a41f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): picked mon.noname-a con 0x7f88a4152cf0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f88a4074020 con 0x7f88a4152cf0 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f88a4073d40 con 0x7f88a4152910 2026-02-22T21:42:48.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f88a4074300 con 0x7f88a4070810 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88aba6d640 10 monclient(hunting): authenticate will time out at 1444.394299s 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 0x7f88a41f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a8fe1640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88a4152910 0x7f88a4165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a97e2640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88a4070810 0x7f88a41f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 0x7f88a41f6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35410/0 (socket says 10.20.193.51:35410) 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 1 -- 10.20.193.51:0/1916659561 learned_addr learned my addr 10.20.193.51:0/1916659561 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 10 monclient(hunting): get_auth_request con 0x7f88a4152cf0 auth_method 0 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a9fe3640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a97e2640 10 monclient(hunting): get_auth_request con 0x7f88a4070810 auth_method 0 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a97e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a97e2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a97e2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a8fe1640 10 monclient(hunting): get_auth_request con 0x7f88a4152910 auth_method 0 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a8fe1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a8fe1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.814+0000 7f88a8fe1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a97e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a97e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a97e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.815+0000 7f88a8fe1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 10 monclient(hunting): handle_auth_done global_id 5211 payload 824 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 -- 10.20.193.51:0/1916659561 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88a4070810 msgr2=0x7f88a41f4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f88a4070810 0x7f88a41f4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 -- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88a4152910 msgr2=0x7f88a4165850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f88a4152910 0x7f88a4165850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 monclient: found mon.noname-a 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.816+0000 7f88a9fe3640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a41644d0 con 0x7f88a4152cf0 2026-02-22T21:42:48.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.817+0000 7f88a9fe3640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 0x7f88a41f6330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88a002f7a0 tx=0x7f88a0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.817+0000 7f8891ffb640 10 client.?.objecter ms_handle_connect 0x7f88a4152cf0 2026-02-22T21:42:48.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.817+0000 7f8891ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:48.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0046020 con 0x7f88a4152cf0 2026-02-22T21:42:48.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: dump: 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.836 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a0041450 con 0x7f88a4152cf0 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 5 monclient: authenticate success, global_id 5211 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0009510 con 0x7f88a4152cf0 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f8891ffb640 10 monclient: dump: 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.837 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88937fe640 4 set_mon_vals no callback set 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 monclient: _renew_subs 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.838 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a41647b0 con 0x7f88a4152cf0 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 client.5211.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 monclient: _renew_subs 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a4164d70 con 0x7f88a4152cf0 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 1 librados: init done 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 client.5211.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 15 client.5211.objecter _session_command_op_assign -1 1 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 20 client.5211.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 client.5211.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.819+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88a4164280 con 0x7f88a4152cf0 2026-02-22T21:42:48.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.831+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f88a0041820 con 0x7f88a4152cf0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.831+0000 7f8891ffb640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8874078e50 0x7f887407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.832+0000 7f88a97e2640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8874078e50 0x7f887407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.832+0000 7f88a97e2640 10 monclient: get_auth_request con 0x7f8874078e50 auth_method 0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.832+0000 7f88a97e2640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8874078e50 0x7f887407b310 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88940051f0 tx=0x7f8894007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.832+0000 7f8891ffb640 10 client.5211.objecter ms_handle_connect 0x7f8874078e50 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f88a0083860 con 0x7f88a4152cf0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 10 client.5211.objecter ms_dispatch2 0x7f88a4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 3 client.5211.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 3 client.5211.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 10 client.5211.objecter checking command 1 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7f8874081330 0x7f8874083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter _get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 15 client.5211.objecter _session_command_op_remove -1 1 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter _get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 15 client.5211.objecter _session_command_op_assign 1 1 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 4 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 10 client.5211.objecter _send_command 1 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88a0044e50 con 0x7f8874081330 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 20 client.5211.objecter dump_active .. 0 homeless 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f88a0083c20 con 0x7f88a4152cf0 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f8891ffb640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.833+0000 7f88a8fe1640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7f8874081330 0x7f8874083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.834+0000 7f88a8fe1640 10 monclient: get_auth_request con 0x7f8874081330 auth_method 0 2026-02-22T21:42:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.835+0000 7f8891ffb640 10 client.5211.objecter ms_handle_connect 0x7f8874081330 2026-02-22T21:42:48.842 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.835+0000 7f88a8fe1640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7f8874081330 0x7f8874083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.845+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== osd.1 v2:10.20.193.68:6824/2540990948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f88a0044e50 con 0x7f8874081330 2026-02-22T21:42:48.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.845+0000 7f8891ffb640 10 client.5211.objecter ms_dispatch2 0x7f88a4000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:48.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.845+0000 7f8891ffb640 10 client.5211.objecter _finish_command 1 = system:0 2026-02-22T21:42:48.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.845+0000 7f8891ffb640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 4 2026-02-22T21:42:48.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.845+0000 7f8891ffb640 15 client.5211.objecter _session_command_op_remove 1 1 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 10 client.5211.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 15 client.5211.objecter _session_command_op_assign -1 2 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 20 client.5211.objecter _get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 20 client.5211.objecter _get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 15 client.5211.objecter _session_command_op_remove -1 2 2026-02-22T21:42:48.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 20 client.5211.objecter get_session s=0x7f8874080c60 osd=1 3 2026-02-22T21:42:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 15 client.5211.objecter _session_command_op_assign 1 2 2026-02-22T21:42:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 4 2026-02-22T21:42:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 10 client.5211.objecter _send_command 2 2026-02-22T21:42:48.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.851+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 --> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f88a40745e0 con 0x7f8874081330 2026-02-22T21:42:48.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f8891ffb640 1 -- 10.20.193.51:0/1916659561 <== osd.1 v2:10.20.193.68:6824/2540990948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f88a40745e0 con 0x7f8874081330 2026-02-22T21:42:48.866 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f8891ffb640 10 client.5211.objecter ms_dispatch2 0x7f88a4000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f8891ffb640 10 client.5211.objecter _finish_command 2 = system:0 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f8891ffb640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 4 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f8891ffb640 15 client.5211.objecter _session_command_op_remove 1 2 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:48.867 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 10 librados: watch_flush enter 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 10 librados: watch_flush exit 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 10 client.5211.objecter close_session for osd.1 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7f8874081330 msgr2=0x7f8874083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.68:6824/2540990948,v1:10.20.193.68:6825/2540990948] conn(0x7f8874081330 0x7f8874083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 20 client.5211.objecter put_session s=0x7f8874080c60 osd=1 1 2026-02-22T21:42:48.869 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 20 client.5211.objecter shutdown clearing up homeless session... 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 10 client.5211.objecter successfully canceled tick 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8874078e50 msgr2=0x7f887407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8874078e50 0x7f887407b310 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88940051f0 tx=0x7f8894007f30 comp rx=0 tx=0).stop 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 10 monclient: shutdown 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 msgr2=0x7f88a41f6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 --2- 10.20.193.51:0/1916659561 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f88a4152cf0 0x7f88a41f6330 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88a002f7a0 tx=0x7f88a0004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88a9fe3640 1 -- 10.20.193.51:0/1916659561 reap_dead start 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 shutdown_connections 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 >> 10.20.193.51:0/1916659561 conn(0x7f88a40c9d50 msgr2=0x7f88a40c65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 shutdown_connections 2026-02-22T21:42:48.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 -- 10.20.193.51:0/1916659561 wait complete. 2026-02-22T21:42:48.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.865+0000 7f88aba6d640 1 librados: shutdown 2026-02-22T21:42:48.875 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-22T21:42:48.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.941+0000 7f22423d3640 10 monclient: get_monmap_and_config 2026-02-22T21:42:48.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.941+0000 7f22423d3640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.941+0000 7f22423d3640 10 monclient: monmap: 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:48.942169+0000 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:48.942169+0000 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:48.947 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.941+0000 7f22423d3640 1 Processor -- start 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 1 -- start start 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient: init 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient: _add_conns ranks=[4,2,1,0,3,5] 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f223c150220 0x7f223c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-b con 0x7f223c150220 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f223c150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:48.948 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f223c15c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f223c15a920 con 0x7f223c15c710 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f223c15af60 con 0x7f223c150600 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f223c15b240 con 0x7f223c150220 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.942+0000 7f22423d3640 10 monclient(hunting): authenticate will time out at 1444.522564s 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223b7fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f223c150600 0x7f223c15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47580/0 (socket says 10.20.193.51:47580) 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223b7fe640 1 -- 10.20.193.51:0/3871669782 learned_addr learned my addr 10.20.193.51:0/3871669782 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 1 --2- 10.20.193.51:0/3871669782 >> v2:10.20.193.68:3300/0 conn(0x7f223c150220 0x7f223c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): get_auth_request con 0x7f223c150220 auth_method 0 2026-02-22T21:42:48.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.943+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 1 -- 10.20.193.51:0/3871669782 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573793918 0 0) 0x7f223c15af60 con 0x7f223c150600 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 10 monclient(hunting): my global_id is 5204 2026-02-22T21:42:48.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223affd640 1 -- 10.20.193.51:0/3871669782 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2220003620 con 0x7f223c150600 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 10 monclient(hunting): handle_auth_done global_id 5212 payload 274 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 1 -- 10.20.193.51:0/3871669782 >> v1:10.20.193.125:6789/0 conn(0x7f223c150600 legacy=0x7f223c15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 1 -- 10.20.193.51:0/3871669782 >> v1:10.20.193.68:6789/0 conn(0x7f223c15c710 legacy=0x7f223c06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 1 monclient: found mon.noname-b 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.944+0000 7f223bfff640 1 -- 10.20.193.51:0/3871669782 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c15b520 con 0x7f223c150220 2026-02-22T21:42:48.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.946+0000 7f223bfff640 1 --2- 10.20.193.51:0/3871669782 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c0c73a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22280098a0 tx=0x7f222802ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73cf444309d9c367 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 1 -- 10.20.193.51:0/3871669782 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222803c070 con 0x7f223c150220 2026-02-22T21:42:48.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: dump: 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.963 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f22423d3640 5 monclient: authenticate success, global_id 5212 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f22423d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 1 -- 10.20.193.51:0/3871669782 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222802fc10 con 0x7f223c150220 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 1 -- 10.20.193.51:0/3871669782 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228037440 con 0x7f223c150220 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:48.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f223affd640 10 monclient: dump: 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f22423d3640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:48.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.948+0000 7f22423d3640 4 set_mon_vals no callback set 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 10 monclient: shutdown 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- 10.20.193.51:0/3871669782 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 msgr2=0x7f223c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 --2- 10.20.193.51:0/3871669782 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c0c73a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22280098a0 tx=0x7f222802ef00 comp rx=0 tx=0).stop 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- 10.20.193.51:0/3871669782 shutdown_connections 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 --2- 10.20.193.51:0/3871669782 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c0c73a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- 10.20.193.51:0/3871669782 >> 10.20.193.51:0/3871669782 conn(0x7f223c0cd7b0 msgr2=0x7f223c0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- 10.20.193.51:0/3871669782 shutdown_connections 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- 10.20.193.51:0/3871669782 wait complete. 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 10 monclient: build_initial_monmap 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 10 monclient: monmap: 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:48.968 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 librados: starting msgr at 2026-02-22T21:42:48.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 librados: starting objecter 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 Processor -- start 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 -- start start 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 librados: setting wanted keys 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 1 librados: calling monclient init 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.949+0000 7f22423d3640 10 monclient: init 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:48.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-b con 0x7f223c150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f223c150600 0x7f223c1564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-a con 0x7f223c150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f223c15c710 0x7f223c156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): picked mon.noname-c con 0x7f223c15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:48.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f223c0703b0 con 0x7f223c150600 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f223c15bae0 con 0x7f223c150220 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f223c070530 con 0x7f223c15c710 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:48.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f22423d3640 10 monclient(hunting): authenticate will time out at 1444.530202s 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f223c150600 0x7f223c1564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f2240949640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f223c15c710 0x7f223c156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:40988/0 (socket says 10.20.193.51:40988) 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 1 -- 10.20.193.51:0/708699120 learned_addr learned my addr 10.20.193.51:0/708699120 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:48.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): get_auth_request con 0x7f223c150220 auth_method 0 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): get_auth_request con 0x7f223c150600 auth_method 0 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f2240949640 10 monclient(hunting): get_auth_request con 0x7f223c15c710 auth_method 0 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f2240949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f2240949640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f2240949640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.950+0000 7f223b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.951+0000 7f223b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.951+0000 7f2240949640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.951+0000 7f2240949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.951+0000 7f2240949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 10 monclient(hunting): handle_auth_done global_id 5218 payload 824 2026-02-22T21:42:48.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 -- 10.20.193.51:0/708699120 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f223c15c710 msgr2=0x7f223c156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f223c15c710 0x7f223c156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 -- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f223c150600 msgr2=0x7f223c1564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f223c150600 0x7f223c1564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 monclient: found mon.noname-b 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:48.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.952+0000 7f223bfff640 1 -- 10.20.193.51:0/708699120 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c1f2900 con 0x7f223c150220 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.953+0000 7f223bfff640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f222803a040 tx=0x7f22280047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.953+0000 7f2238ff9640 10 client.?.objecter ms_handle_connect 0x7f223c150220 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.953+0000 7f2238ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222803c040 con 0x7f223c150220 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: dump: 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.979 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 5 monclient: authenticate success, global_id 5218 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 10 monclient: _renew_subs 2026-02-22T21:42:48.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c1f1aa0 con 0x7f223c150220 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 10 client.5218.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 10 monclient: _renew_subs 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c1f2060 con 0x7f223c150220 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f22423d3640 1 librados: init done 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2228003b50 con 0x7f223c150220 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:48.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228037440 con 0x7f223c150220 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f2238ff9640 10 monclient: dump: 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:48.982 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:48.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.954+0000 7f223a7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:48.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.955+0000 7f22423d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.955+0000 7f22423d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:48.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.955+0000 7f22423d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:48.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.955+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223c156220 con 0x7f223c150220 2026-02-22T21:42:48.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.966+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f22280375e0 con 0x7f223c150220 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.966+0000 7f2238ff9640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 0x7f220c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.966+0000 7f223b7fe640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 0x7f220c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.966+0000 7f223b7fe640 10 monclient: get_auth_request con 0x7f220c078e00 auth_method 0 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.967+0000 7f2238ff9640 10 client.5218.objecter ms_handle_connect 0x7f220c078e00 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.967+0000 7f223b7fe640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 0x7f220c07b2c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f222c007b60 tx=0x7f222c007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.968+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f2228083560 con 0x7f223c150220 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.968+0000 7f2238ff9640 10 client.5218.objecter ms_dispatch2 0x7f223c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.968+0000 7f2238ff9640 3 client.5218.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.968+0000 7f2238ff9640 3 client.5218.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:48.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:48.968+0000 7f2238ff9640 20 client.5218.objecter dump_active .. 0 homeless 2026-02-22T21:42:49.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.071+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f2228087520 con 0x7f223c150220 2026-02-22T21:42:49.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.071+0000 7f2238ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.071+0000 7f2238ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:49.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.155+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f223c0c60c0 con 0x7f220c078e00 2026-02-22T21:42:49.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.155+0000 7f2238ff9640 1 -- 10.20.193.51:0/708699120 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f223c0c60c0 con 0x7f220c078e00 2026-02-22T21:42:49.157 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T21:42:49.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 10 librados: watch_flush enter 2026-02-22T21:42:49.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 10 librados: watch_flush exit 2026-02-22T21:42:49.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 20 client.5218.objecter shutdown clearing up homeless session... 2026-02-22T21:42:49.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 10 client.5218.objecter successfully canceled tick 2026-02-22T21:42:49.159 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 msgr2=0x7f220c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 0x7f220c07b2c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f222c007b60 tx=0x7f222c007ab0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 10 monclient: shutdown 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 msgr2=0x7f223c14ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f222803a040 tx=0x7f22280047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 shutdown_connections 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f220c078e00 0x7f220c07b2c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f223c15c710 0x7f223c156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f223c150600 0x7f223c1564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 --2- 10.20.193.51:0/708699120 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f223c150220 0x7f223c14ca50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 >> 10.20.193.51:0/708699120 conn(0x7f223c0cd7b0 msgr2=0x7f223c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 shutdown_connections 2026-02-22T21:42:49.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 -- 10.20.193.51:0/708699120 wait complete. 2026-02-22T21:42:49.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.156+0000 7f22423d3640 1 librados: shutdown 2026-02-22T21:42:49.167 INFO:tasks.ceph:Scrubbing osd.5 2026-02-22T21:42:49.167 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:49.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 10 monclient: get_monmap_and_config 2026-02-22T21:42:49.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.237 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 10 monclient: monmap: 2026-02-22T21:42:49.237 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.237 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:49.235865+0000 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:49.235865+0000 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:49.238 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 1 Processor -- start 2026-02-22T21:42:49.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 1 -- start start 2026-02-22T21:42:49.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.235+0000 7f54e2572640 10 monclient: init 2026-02-22T21:42:49.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient: _add_conns ranks=[1,0,2,5,4,3] 2026-02-22T21:42:49.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f54dc153860 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54dc0c9500 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54dc0c7c50 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54dc0c8180 con 0x7f54dc0c9500 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54dc0c8460 con 0x7f54dc153860 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54dc0c8740 con 0x7f54dc0c7c50 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54e2572640 10 monclient(hunting): authenticate will time out at 1444.816210s 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54db7fe640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f54dc0c9500 0x7f54dc15d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58968/0 (socket says 10.20.193.51:58968) 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.236+0000 7f54db7fe640 1 -- 10.20.193.51:0/2512743121 learned_addr learned my addr 10.20.193.51:0/2512743121 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144855200 0 0) 0x7f54dc0c8180 con 0x7f54dc0c9500 2026-02-22T21:42:49.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): my global_id is 5217 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54c0003620 con 0x7f54dc0c9500 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3723459188 0 0) 0x7f54dc0c8460 con 0x7f54dc153860 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): my global_id is 5221 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54dc0c8180 con 0x7f54dc153860 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 451462199 0 0) 0x7f54dc0c8740 con 0x7f54dc0c7c50 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _negotiate 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 10 monclient(hunting): my global_id is 5210 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.237+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54dc0c8460 con 0x7f54dc0c7c50 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.0 v1:10.20.193.51:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1268454354 0 0) 0x7f54c0003620 con 0x7f54dc0c9500 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 >> v1:10.20.193.125:6789/0 conn(0x7f54dc0c7c50 legacy=0x7f54dc1643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 >> v1:10.20.193.68:6789/0 conn(0x7f54dc153860 legacy=0x7f54dc15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 monclient: found mon.noname-a-legacy 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:10.20.193.51:6789/0 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> v1:10.20.193.51:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc0c8a20 con 0x7f54dc0c9500 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.0 v1:10.20.193.51:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f54cc002f00 con 0x7f54dc0c9500 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: dump: 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.256 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 monclient: mon.a has (v2) addrs [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] but i'm connected to v1:10.20.193.51:6789/0, reconnecting 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 >> v1:10.20.193.51:6789/0 conn(0x7f54dc0c9500 legacy=0x7f54dc15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:49.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0003fe0 0x7f54c0004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): picked mon.b con 0x7f54c0003fe0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54c0004e40 0x7f54c00052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): picked mon.a con 0x7f54c0004e40 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 0x7f54c000bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): picked mon.c con 0x7f54c0009ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f54c000c570 con 0x7f54c0004e40 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f54c000c6f0 con 0x7f54c0003fe0 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f54c000c8f0 con 0x7f54c0009ab0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.239+0000 7f54daffd640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0003fe0 0x7f54c0004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 0x7f54c000bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): get_auth_request con 0x7f54c0009ab0 auth_method 0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): get_auth_request con 0x7f54c0003fe0 auth_method 0 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:49.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.240+0000 7f54db7fe640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54c0004e40 0x7f54c00052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): get_auth_request con 0x7f54c0004e40 auth_method 0 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:49.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_done global_id 5217 payload 293 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 -- 10.20.193.51:0/2512743121 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0003fe0 msgr2=0x7f54c0004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0003fe0 0x7f54c0004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 -- 10.20.193.51:0/2512743121 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54c0004e40 msgr2=0x7f54c00052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54c0004e40 0x7f54c00052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 monclient: found mon.c 2026-02-22T21:42:49.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 -- 10.20.193.51:0/2512743121 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c0003620 con 0x7f54c0009ab0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e2572640 5 monclient: authenticate success, global_id 5217 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e2572640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.241+0000 7f54e0ae8640 1 -- 10.20.193.51:0/2512743121 reap_dead start 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.242+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 0x7f54c000bea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c320 tx=0x7f54d002f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78313fa1f1ce0e5c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0003860 con 0x7f54c0009ab0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: dump: 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54e2572640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d0003c70 con 0x7f54c0009ab0 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 1 -- 10.20.193.51:0/2512743121 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0005310 con 0x7f54c0009ab0 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54daffd640 10 monclient: dump: 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.264 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54e2572640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.243+0000 7f54e2572640 4 set_mon_vals no callback set 2026-02-22T21:42:49.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 10 monclient: shutdown 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 -- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 msgr2=0x7f54c000bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 0x7f54c000bea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c320 tx=0x7f54d002f610 comp rx=0 tx=0).stop 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 -- 10.20.193.51:0/2512743121 shutdown_connections 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 --2- 10.20.193.51:0/2512743121 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54c0009ab0 0x7f54c000bea0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 -- 10.20.193.51:0/2512743121 >> 10.20.193.51:0/2512743121 conn(0x7f54dc0c9d50 msgr2=0x7f54dc0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 -- 10.20.193.51:0/2512743121 shutdown_connections 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 -- 10.20.193.51:0/2512743121 wait complete. 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 10 monclient: monmap: 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:49.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 librados: starting msgr at 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 librados: starting objecter 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.244+0000 7f54e2572640 1 Processor -- start 2026-02-22T21:42:49.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 -- start start 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 librados: setting wanted keys 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 librados: calling monclient init 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient: init 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0009ab0 0x7f54dc147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-b con 0x7f54c0009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54dc1f3a70 0x7f54dc1f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-c con 0x7f54dc1f3a70 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54dc1f5fc0 0x7f54dc06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): picked mon.noname-a con 0x7f54dc1f5fc0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc153c40 con 0x7f54dc1f5fc0 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc0c8fe0 con 0x7f54c0009ab0 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc153dc0 con 0x7f54dc1f3a70 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e2572640 10 monclient(hunting): authenticate will time out at 1444.825443s 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54dc1f5fc0 0x7f54dc06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54db7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54dc1f3a70 0x7f54dc1f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54dc1f5fc0 0x7f54dc06fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35446/0 (socket says 10.20.193.51:35446) 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 1 -- 10.20.193.51:0/651332522 learned_addr learned my addr 10.20.193.51:0/651332522 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): get_auth_request con 0x7f54dc1f5fc0 auth_method 0 2026-02-22T21:42:49.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54dbfff640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0009ab0 0x7f54dc147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54db7fe640 10 monclient(hunting): get_auth_request con 0x7f54dc1f3a70 auth_method 0 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54db7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54db7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.245+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54e0ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): get_auth_request con 0x7f54c0009ab0 auth_method 0 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.246+0000 7f54dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 10 monclient(hunting): handle_auth_done global_id 5213 payload 824 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 -- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0009ab0 msgr2=0x7f54dc147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f54c0009ab0 0x7f54dc147930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 -- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54dc1f5fc0 msgr2=0x7f54dc06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f54dc1f5fc0 0x7f54dc06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 monclient: found mon.noname-c 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.247+0000 7f54db7fe640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc1f6440 con 0x7f54dc1f3a70 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.248+0000 7f54db7fe640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54dc1f3a70 0x7f54dc1f5a80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0028f0 tx=0x7f54cc007500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.248+0000 7f54d8ff9640 10 client.?.objecter ms_handle_connect 0x7f54dc1f3a70 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.248+0000 7f54d8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc007850 con 0x7f54dc1f3a70 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: dump: 2026-02-22T21:42:49.272 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.273 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 5 monclient: authenticate success, global_id 5213 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 monclient: _renew_subs 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc147e70 con 0x7f54dc1f3a70 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 client.5213.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 monclient: _renew_subs 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc074630 con 0x7f54dc1f3a70 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 1 librados: init done 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54cc018070 con 0x7f54dc1f3a70 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00f980 con 0x7f54dc1f3a70 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54d8ff9640 10 monclient: dump: 2026-02-22T21:42:49.274 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.275 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54da7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 client.5213.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 15 client.5213.objecter _session_command_op_assign -1 1 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 20 client.5213.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 client.5213.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:49.276 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.249+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f54dc1f4920 con 0x7f54dc1f3a70 2026-02-22T21:42:49.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.261+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f54cc00fb20 con 0x7f54dc1f3a70 2026-02-22T21:42:49.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.261+0000 7f54d8ff9640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f54ac078e30 0x7f54ac07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.261+0000 7f54dbfff640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f54ac078e30 0x7f54ac07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.261+0000 7f54dbfff640 10 monclient: get_auth_request con 0x7f54ac078e30 auth_method 0 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54dbfff640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f54ac078e30 0x7f54ac07b2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54c8002910 tx=0x7f54c8007da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54d8ff9640 10 client.5213.objecter ms_handle_connect 0x7f54ac078e30 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f54cc00fe40 con 0x7f54dc1f3a70 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54d8ff9640 10 client.5213.objecter ms_dispatch2 0x7f54dc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54d8ff9640 3 client.5213.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.262+0000 7f54d8ff9640 3 client.5213.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 10 client.5213.objecter checking command 1 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7f54ac081310 0x7f54ac083730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter _get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 15 client.5213.objecter _session_command_op_remove -1 1 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter _get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 15 client.5213.objecter _session_command_op_assign 5 1 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 4 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 10 client.5213.objecter _send_command 1 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54ac083e00 con 0x7f54ac081310 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 20 client.5213.objecter dump_active .. 0 homeless 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f54cc05e4c0 con 0x7f54dc1f3a70 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54d8ff9640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7f54ac081310 0x7f54ac083730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.263+0000 7f54e0ae8640 10 monclient: get_auth_request con 0x7f54ac081310 auth_method 0 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.265+0000 7f54d8ff9640 10 client.5213.objecter ms_handle_connect 0x7f54ac081310 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.265+0000 7f54e0ae8640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7f54ac081310 0x7f54ac083730 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.276+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== osd.5 v2:10.20.193.68:6816/3736303751 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f54ac083e00 con 0x7f54ac081310 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.276+0000 7f54d8ff9640 10 client.5213.objecter ms_dispatch2 0x7f54dc000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.276+0000 7f54d8ff9640 10 client.5213.objecter _finish_command 1 = system:0 2026-02-22T21:42:49.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.276+0000 7f54d8ff9640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 4 2026-02-22T21:42:49.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.276+0000 7f54d8ff9640 15 client.5213.objecter _session_command_op_remove 5 1 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 10 client.5213.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 15 client.5213.objecter _session_command_op_assign -1 2 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 20 client.5213.objecter _get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 20 client.5213.objecter _get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 15 client.5213.objecter _session_command_op_remove -1 2 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 20 client.5213.objecter get_session s=0x7f54ac080c40 osd=5 3 2026-02-22T21:42:49.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 15 client.5213.objecter _session_command_op_assign 5 2 2026-02-22T21:42:49.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 4 2026-02-22T21:42:49.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 10 client.5213.objecter _send_command 2 2026-02-22T21:42:49.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.281+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 --> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f54dc074910 con 0x7f54ac081310 2026-02-22T21:42:49.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54d8ff9640 1 -- 10.20.193.51:0/651332522 <== osd.5 v2:10.20.193.68:6816/3736303751 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f54d0003a70 con 0x7f54ac081310 2026-02-22T21:42:49.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54d8ff9640 10 client.5213.objecter ms_dispatch2 0x7f54dc000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:49.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54d8ff9640 10 client.5213.objecter _finish_command 2 = system:0 2026-02-22T21:42:49.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54d8ff9640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 4 2026-02-22T21:42:49.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54d8ff9640 15 client.5213.objecter _session_command_op_remove 5 2 2026-02-22T21:42:49.299 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:49.299 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:49.299 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:49.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 10 librados: watch_flush enter 2026-02-22T21:42:49.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 10 librados: watch_flush exit 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 10 client.5213.objecter close_session for osd.5 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7f54ac081310 msgr2=0x7f54ac083730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.68:6816/3736303751,v1:10.20.193.68:6817/3736303751] conn(0x7f54ac081310 0x7f54ac083730 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 20 client.5213.objecter put_session s=0x7f54ac080c40 osd=5 1 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 20 client.5213.objecter shutdown clearing up homeless session... 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 10 client.5213.objecter successfully canceled tick 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f54ac078e30 msgr2=0x7f54ac07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f54ac078e30 0x7f54ac07b2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f54c8002910 tx=0x7f54c8007da0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.302 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 10 monclient: shutdown 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54dc1f3a70 msgr2=0x7f54dc1f5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 --2- 10.20.193.51:0/651332522 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f54dc1f3a70 0x7f54dc1f5a80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0028f0 tx=0x7f54cc007500 comp rx=0 tx=0).stop 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e0ae8640 1 -- 10.20.193.51:0/651332522 reap_dead start 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 shutdown_connections 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 >> 10.20.193.51:0/651332522 conn(0x7f54dc0c9d50 msgr2=0x7f54dc0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 shutdown_connections 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 -- 10.20.193.51:0/651332522 wait complete. 2026-02-22T21:42:49.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.295+0000 7f54e2572640 1 librados: shutdown 2026-02-22T21:42:49.305 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 10 monclient: get_monmap_and_config 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 10 monclient: monmap: 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:49.372716+0000 2026-02-22T21:42:49.374 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:49.372716+0000 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 1 Processor -- start 2026-02-22T21:42:49.375 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 1 -- start start 2026-02-22T21:42:49.376 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.372+0000 7f4830753640 10 monclient: init 2026-02-22T21:42:49.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient: _add_conns ranks=[1,3,5,2,0,4] 2026-02-22T21:42:49.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f4828150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:49.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f482815c710 0x7f48280c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.384 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): picked mon.noname-a con 0x7f482815c710 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4828150600 0x7f482806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): picked mon.noname-c con 0x7f4828150600 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482815a920 con 0x7f4828150220 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f482815af60 con 0x7f482815c710 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f482815b240 con 0x7f4828150600 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f4830753640 10 monclient(hunting): authenticate will time out at 1444.953144s 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f482815c710 0x7f48280c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.385 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482ecc9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f4828150600 0x7f482806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f482815c710 0x7f48280c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35460/0 (socket says 10.20.193.51:35460) 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482e4c8640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f4828150220 0x7f482815c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:38992/0 (socket says 10.20.193.51:38992) 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 1 -- 10.20.193.51:0/1270129487 learned_addr learned my addr 10.20.193.51:0/1270129487 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 10 monclient(hunting): get_auth_request con 0x7f482815c710 auth_method 0 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482dcc7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482ecc9640 10 monclient(hunting): get_auth_request con 0x7f4828150600 auth_method 0 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482ecc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482ecc9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.386 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.373+0000 7f482ecc9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 10 monclient(hunting): handle_auth_done global_id 5223 payload 274 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 1 -- 10.20.193.51:0/1270129487 >> v2:10.20.193.125:3300/0 conn(0x7f4828150600 msgr2=0x7f482806fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 1 --2- 10.20.193.51:0/1270129487 >> v2:10.20.193.125:3300/0 conn(0x7f4828150600 0x7f482806fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 1 -- 10.20.193.51:0/1270129487 >> v1:10.20.193.68:6789/0 conn(0x7f4828150220 legacy=0x7f482815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.387 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.374+0000 7f482dcc7640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.375+0000 7f482dcc7640 1 monclient: found mon.noname-a 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.375+0000 7f482dcc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.375+0000 7f482dcc7640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.375+0000 7f482dcc7640 1 -- 10.20.193.51:0/1270129487 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482815b520 con 0x7f482815c710 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.375+0000 7f482dcc7640 1 --2- 10.20.193.51:0/1270129487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f48280c73a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f481c009970 tx=0x7f481c02efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972e44c06090c7c3 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 1 -- 10.20.193.51:0/1270129487 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c03c070 con 0x7f482815c710 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: dump: 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.388 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.389 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 1 -- 10.20.193.51:0/1270129487 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c02fc80 con 0x7f482815c710 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 1 -- 10.20.193.51:0/1270129487 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c037660 con 0x7f482815c710 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f4830753640 5 monclient: authenticate success, global_id 5223 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f4830753640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f4830753640 4 set_mon_vals no callback set 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.377+0000 7f482d4c6640 10 monclient: dump: 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.390 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 10 monclient: shutdown 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 -- 10.20.193.51:0/1270129487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 msgr2=0x7f48280c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 --2- 10.20.193.51:0/1270129487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f48280c73a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f481c009970 tx=0x7f481c02efb0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 -- 10.20.193.51:0/1270129487 shutdown_connections 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 --2- 10.20.193.51:0/1270129487 >> v2:10.20.193.125:3300/0 conn(0x7f4828150600 0x7f482806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 --2- 10.20.193.51:0/1270129487 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f48280c73a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 -- 10.20.193.51:0/1270129487 >> 10.20.193.51:0/1270129487 conn(0x7f48280cd7b0 msgr2=0x7f48280cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 -- 10.20.193.51:0/1270129487 shutdown_connections 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 -- 10.20.193.51:0/1270129487 wait complete. 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 10 monclient: monmap: 2026-02-22T21:42:49.392 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.378+0000 7f4830753640 1 librados: starting msgr at 2026-02-22T21:42:49.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 librados: starting objecter 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 Processor -- start 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 -- start start 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 librados: setting wanted keys 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 librados: calling monclient init 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient: init 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4828150220 0x7f4828154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): picked mon.noname-c con 0x7f4828150220 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4828150600 0x7f48281f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): picked mon.noname-b con 0x7f4828150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:49.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): picked mon.noname-a con 0x7f482815c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f48280701e0 con 0x7f482815c710 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f482815bae0 con 0x7f4828150600 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f4828070360 con 0x7f4828150220 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f4830753640 10 monclient(hunting): authenticate will time out at 1444.959618s 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35464/0 (socket says 10.20.193.51:35464) 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 1 -- 10.20.193.51:0/1498930804 learned_addr learned my addr 10.20.193.51:0/1498930804 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 10 monclient(hunting): get_auth_request con 0x7f482815c710 auth_method 0 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482ecc9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.379+0000 7f482dcc7640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4828150600 0x7f48281f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482ecc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): get_auth_request con 0x7f4828150600 auth_method 0 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482dcc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.380+0000 7f482e4c8640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4828150220 0x7f4828154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482e4c8640 10 monclient(hunting): get_auth_request con 0x7f4828150220 auth_method 0 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482e4c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482e4c8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482e4c8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 10 monclient(hunting): handle_auth_done global_id 5226 payload 824 2026-02-22T21:42:49.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 -- 10.20.193.51:0/1498930804 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4828150220 msgr2=0x7f4828154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4828150220 0x7f4828154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 -- 10.20.193.51:0/1498930804 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4828150600 msgr2=0x7f48281f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4828150600 0x7f48281f2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 monclient: found mon.noname-a 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.381+0000 7f482ecc9640 1 -- 10.20.193.51:0/1498930804 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48281f27b0 con 0x7f482815c710 2026-02-22T21:42:49.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.384+0000 7f482ecc9640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4824009dd0 tx=0x7f482400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.384+0000 7f480effd640 10 client.?.objecter ms_handle_connect 0x7f482815c710 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.384+0000 7f480effd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824018020 con 0x7f482815c710 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: dump: 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.402 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4824014480 con 0x7f482815c710 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824014870 con 0x7f482815c710 2026-02-22T21:42:49.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f480effd640 10 monclient: dump: 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f482ccc5640 4 set_mon_vals no callback set 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 5 monclient: authenticate success, global_id 5226 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _renew_subs 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48281e6060 con 0x7f482815c710 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 client.5226.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _renew_subs 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48281e6620 con 0x7f482815c710 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 1 librados: init done 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.386+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f482806fca0 con 0x7f482815c710 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.398+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f4824014a10 con 0x7f482815c710 2026-02-22T21:42:49.405 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.398+0000 7f480effd640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 0x7f47f807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.398+0000 7f482e4c8640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 0x7f47f807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.398+0000 7f482e4c8640 10 monclient: get_auth_request con 0x7f47f8078e50 auth_method 0 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 10 client.5226.objecter ms_handle_connect 0x7f47f8078e50 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f482e4c8640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 0x7f47f807b310 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4818007fd0 tx=0x7f4818007da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f482405eed0 con 0x7f482815c710 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 10 client.5226.objecter ms_dispatch2 0x7f4828000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 3 client.5226.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 3 client.5226.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:49.406 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.399+0000 7f480effd640 20 client.5226.objecter dump_active .. 0 homeless 2026-02-22T21:42:49.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.500+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f4824022070 con 0x7f482815c710 2026-02-22T21:42:49.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.500+0000 7f480effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.502 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.500+0000 7f480effd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:49.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f4828074da0 con 0x7f47f8078e50 2026-02-22T21:42:49.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f480effd640 1 -- 10.20.193.51:0/1498930804 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4828074da0 con 0x7f47f8078e50 2026-02-22T21:42:49.586 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 5 to deep-scrub 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f4830753640 10 librados: watch_flush enter 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f4830753640 10 librados: watch_flush exit 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f4830753640 20 client.5226.objecter shutdown clearing up homeless session... 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.584+0000 7f4830753640 10 client.5226.objecter successfully canceled tick 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 msgr2=0x7f47f807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.588 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 0x7f47f807b310 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4818007fd0 tx=0x7f4818007da0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 10 monclient: shutdown 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 msgr2=0x7f482814c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4824009dd0 tx=0x7f482400c490 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 shutdown_connections 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f47f8078e50 0x7f47f807b310 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f482815c710 0x7f482814c470 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f4828150600 0x7f48281f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 --2- 10.20.193.51:0/1498930804 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f4828150220 0x7f4828154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 >> 10.20.193.51:0/1498930804 conn(0x7f48280cd7b0 msgr2=0x7f48280cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 shutdown_connections 2026-02-22T21:42:49.589 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 -- 10.20.193.51:0/1498930804 wait complete. 2026-02-22T21:42:49.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.585+0000 7f4830753640 1 librados: shutdown 2026-02-22T21:42:49.595 INFO:tasks.ceph:Scrubbing osd.9 2026-02-22T21:42:49.595 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 10 monclient: monmap: 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.666 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:49.666059+0000 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:49.666059+0000 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 1 Processor -- start 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 1 -- start start 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.665+0000 7f078963c640 10 monclient: init 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient: _add_conns ranks=[4,5,2,1,3,0] 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f0784153860 0x7f078415af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): picked mon.noname-b con 0x7f0784153860 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f07840c9500 0x7f078415d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): picked mon.noname-c con 0x7f07840c9500 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07840c7c50 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07840c8180 con 0x7f07840c7c50 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f07840c8460 con 0x7f0784153860 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f07840c8740 con 0x7f07840c9500 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f078963c640 10 monclient(hunting): authenticate will time out at 1445.246371s 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f0782ffd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f0784153860 0x7f078415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f07837fe640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f07840c7c50 0x7f0784159a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47604/0 (socket says 10.20.193.51:47604) 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.666+0000 7f07837fe640 1 -- 10.20.193.51:0/2124056425 learned_addr learned my addr 10.20.193.51:0/2124056425 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): get_auth_request con 0x7f0784153860 auth_method 0 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 1 --2- 10.20.193.51:0/2124056425 >> v2:10.20.193.125:3300/0 conn(0x7f07840c9500 0x7f078415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): get_auth_request con 0x7f07840c9500 auth_method 0 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.667+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 10 monclient(hunting): handle_auth_done global_id 5233 payload 274 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 1 -- 10.20.193.51:0/2124056425 >> v2:10.20.193.125:3300/0 conn(0x7f07840c9500 msgr2=0x7f078415d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 1 --2- 10.20.193.51:0/2124056425 >> v2:10.20.193.125:3300/0 conn(0x7f07840c9500 0x7f078415d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 1 -- 10.20.193.51:0/2124056425 >> v1:10.20.193.125:6789/0 conn(0x7f07840c7c50 legacy=0x7f0784159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 1 monclient: found mon.noname-b 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:49.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.668+0000 7f0782ffd640 1 -- 10.20.193.51:0/2124056425 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07840c8a20 con 0x7f0784153860 2026-02-22T21:42:49.670 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.669+0000 7f0782ffd640 1 --2- 10.20.193.51:0/2124056425 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784153860 0x7f078415af10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f078415cf50 tx=0x7f077802ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab17d0462478c2ec server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 1 -- 10.20.193.51:0/2124056425 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077803c070 con 0x7f0784153860 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: dump: 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.672 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 5 monclient: authenticate success, global_id 5233 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 1 -- 10.20.193.51:0/2124056425 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077802faf0 con 0x7f0784153860 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 1 -- 10.20.193.51:0/2124056425 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778037440 con 0x7f0784153860 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 4 set_mon_vals no callback set 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f0781ffb640 10 monclient: dump: 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.673 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 10 monclient: shutdown 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 -- 10.20.193.51:0/2124056425 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784153860 msgr2=0x7f078415af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 --2- 10.20.193.51:0/2124056425 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784153860 0x7f078415af10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f078415cf50 tx=0x7f077802ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 -- 10.20.193.51:0/2124056425 shutdown_connections 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 --2- 10.20.193.51:0/2124056425 >> v2:10.20.193.125:3300/0 conn(0x7f07840c9500 0x7f078415d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 --2- 10.20.193.51:0/2124056425 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784153860 0x7f078415af10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 -- 10.20.193.51:0/2124056425 >> 10.20.193.51:0/2124056425 conn(0x7f07840c9d50 msgr2=0x7f07840ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 -- 10.20.193.51:0/2124056425 shutdown_connections 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.671+0000 7f078963c640 1 -- 10.20.193.51:0/2124056425 wait complete. 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient: monmap: 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 librados: starting msgr at 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 librados: starting objecter 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 Processor -- start 2026-02-22T21:42:49.674 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 -- start start 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 librados: setting wanted keys 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 librados: calling monclient init 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient: init 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07840c7c50 0x7f078406edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): picked mon.noname-c con 0x7f07840c7c50 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0784153860 0x7f0784073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): picked mon.noname-a con 0x7f0784153860 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 0x7f0784074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): picked mon.noname-b con 0x7f0784149d70 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f078415b450 con 0x7f0784153860 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f07840c8fe0 con 0x7f0784149d70 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f0784153c40 con 0x7f07840c7c50 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.672+0000 7f078963c640 10 monclient(hunting): authenticate will time out at 1445.252951s 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 0x7f0784074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0784153860 0x7f0784073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 0x7f0784074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:41016/0 (socket says 10.20.193.51:41016) 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 1 -- 10.20.193.51:0/3949058151 learned_addr learned my addr 10.20.193.51:0/3949058151 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): get_auth_request con 0x7f0784149d70 auth_method 0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f0782ffd640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07840c7c50 0x7f078406edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): get_auth_request con 0x7f0784153860 auth_method 0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f0782ffd640 10 monclient(hunting): get_auth_request con 0x7f07840c7c50 auth_method 0 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f0782ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f0782ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f0782ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.673+0000 7f07827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.674+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.674+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.674+0000 7f0782ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 10 monclient(hunting): handle_auth_done global_id 5236 payload 824 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 -- 10.20.193.51:0/3949058151 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07840c7c50 msgr2=0x7f078406edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f07840c7c50 0x7f078406edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 -- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0784153860 msgr2=0x7f0784073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f0784153860 0x7f0784073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 monclient: found mon.noname-b 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.675+0000 7f07837fe640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07841425a0 con 0x7f0784149d70 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.676+0000 7f07837fe640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 0x7f0784074160 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f077400cad0 tx=0x7f077400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.676+0000 7f07637fe640 10 client.?.objecter ms_handle_connect 0x7f0784149d70 2026-02-22T21:42:49.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.676+0000 7f07637fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774017070 con 0x7f0784149d70 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: dump: 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077400ac70 con 0x7f0784149d70 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774012640 con 0x7f0784149d70 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07637fe640 10 monclient: dump: 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.679 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 5 monclient: authenticate success, global_id 5236 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 10 monclient: _renew_subs 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f07817fa640 4 set_mon_vals no callback set 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078406f330 con 0x7f0784149d70 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 10 client.5236.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 10 monclient: _renew_subs 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078406e030 con 0x7f0784149d70 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.677+0000 7f078963c640 1 librados: init done 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 10 client.5236.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 15 client.5236.objecter _session_command_op_assign -1 1 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 20 client.5236.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 10 client.5236.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:49.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.678+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07840746a0 con 0x7f0784149d70 2026-02-22T21:42:49.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f077401b3b0 con 0x7f0784149d70 2026-02-22T21:42:49.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f07637fe640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0750078e50 0x7f075007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f0782ffd640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0750078e50 0x7f075007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f0782ffd640 10 monclient: get_auth_request con 0x7f0750078e50 auth_method 0 2026-02-22T21:42:49.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f0782ffd640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0750078e50 0x7f075007b310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0778039000 tx=0x7f0778033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.690+0000 7f07637fe640 10 client.5236.objecter ms_handle_connect 0x7f0750078e50 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.691+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f077405df00 con 0x7f0784149d70 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.691+0000 7f07637fe640 10 client.5236.objecter ms_dispatch2 0x7f0784000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.691+0000 7f07637fe640 3 client.5236.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.691+0000 7f07637fe640 3 client.5236.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 10 client.5236.objecter checking command 1 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f0750081390 0x7f07500837b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter _get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 15 client.5236.objecter _session_command_op_remove -1 1 2026-02-22T21:42:49.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter _get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 15 client.5236.objecter _session_command_op_assign 9 1 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 4 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 10 client.5236.objecter _send_command 1 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0750083e80 con 0x7f0750081390 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 20 client.5236.objecter dump_active .. 0 homeless 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f077405e2c0 con 0x7f0784149d70 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07637fe640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07827fc640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f0750081390 0x7f07500837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.692+0000 7f07827fc640 10 monclient: get_auth_request con 0x7f0750081390 auth_method 0 2026-02-22T21:42:49.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.693+0000 7f07637fe640 10 client.5236.objecter ms_handle_connect 0x7f0750081390 2026-02-22T21:42:49.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.693+0000 7f07827fc640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f0750081390 0x7f07500837b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.703+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== osd.9 v2:10.20.193.68:6808/1841964582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f0750083e80 con 0x7f0750081390 2026-02-22T21:42:49.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.703+0000 7f07637fe640 10 client.5236.objecter ms_dispatch2 0x7f0784000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:49.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.703+0000 7f07637fe640 10 client.5236.objecter _finish_command 1 = system:0 2026-02-22T21:42:49.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.703+0000 7f07637fe640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 4 2026-02-22T21:42:49.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.703+0000 7f07637fe640 15 client.5236.objecter _session_command_op_remove 9 1 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 10 client.5236.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 15 client.5236.objecter _session_command_op_assign -1 2 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 20 client.5236.objecter _get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 20 client.5236.objecter _get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 15 client.5236.objecter _session_command_op_remove -1 2 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 20 client.5236.objecter get_session s=0x7f0750080cc0 osd=9 3 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 15 client.5236.objecter _session_command_op_assign 9 2 2026-02-22T21:42:49.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 4 2026-02-22T21:42:49.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 10 client.5236.objecter _send_command 2 2026-02-22T21:42:49.712 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.709+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 --> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f078415af10 con 0x7f0750081390 2026-02-22T21:42:49.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07637fe640 1 -- 10.20.193.51:0/3949058151 <== osd.9 v2:10.20.193.68:6808/1841964582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f078415af10 con 0x7f0750081390 2026-02-22T21:42:49.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07637fe640 10 client.5236.objecter ms_dispatch2 0x7f0784000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:49.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07637fe640 10 client.5236.objecter _finish_command 2 = system:0 2026-02-22T21:42:49.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07637fe640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 4 2026-02-22T21:42:49.725 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07637fe640 15 client.5236.objecter _session_command_op_remove 9 2 2026-02-22T21:42:49.726 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:49.726 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:49.726 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:49.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 10 librados: watch_flush enter 2026-02-22T21:42:49.727 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 10 librados: watch_flush exit 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 10 client.5236.objecter close_session for osd.9 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f0750081390 msgr2=0x7f07500837b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:6808/1841964582,v1:10.20.193.68:6809/1841964582] conn(0x7f0750081390 0x7f07500837b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 20 client.5236.objecter put_session s=0x7f0750080cc0 osd=9 1 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 20 client.5236.objecter shutdown clearing up homeless session... 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 10 client.5236.objecter successfully canceled tick 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0750078e50 msgr2=0x7f075007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0750078e50 0x7f075007b310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0778039000 tx=0x7f0778033000 comp rx=0 tx=0).stop 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 10 monclient: shutdown 2026-02-22T21:42:49.728 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 msgr2=0x7f0784074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 --2- 10.20.193.51:0/3949058151 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f0784149d70 0x7f0784074160 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f077400cad0 tx=0x7f077400a060 comp rx=0 tx=0).stop 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f07837fe640 1 -- 10.20.193.51:0/3949058151 reap_dead start 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 shutdown_connections 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 >> 10.20.193.51:0/3949058151 conn(0x7f07840c9d50 msgr2=0x7f07840ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 shutdown_connections 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 -- 10.20.193.51:0/3949058151 wait complete. 2026-02-22T21:42:49.729 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.724+0000 7f078963c640 1 librados: shutdown 2026-02-22T21:42:49.732 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: get_monmap_and_config 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: monmap: 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:49.800717+0000 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:49.800717+0000 2026-02-22T21:42:49.802 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 1 Processor -- start 2026-02-22T21:42:49.803 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 1 -- start start 2026-02-22T21:42:49.804 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: init 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient: _add_conns ranks=[3,1,2,5,4,0] 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f09cc150220 0x7f09cc0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.800+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f09cc150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09cc15c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc15a920 con 0x7f09cc150600 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc15af60 con 0x7f09cc15c710 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f09cc15b240 con 0x7f09cc150220 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09d2a61640 10 monclient(hunting): authenticate will time out at 1445.381047s 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f09cc150220 0x7f09cc0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f09cc150220 0x7f09cc0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35482/0 (socket says 10.20.193.51:35482) 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 1 -- 10.20.193.51:0/307256701 learned_addr learned my addr 10.20.193.51:0/307256701 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): get_auth_request con 0x7f09cc150220 auth_method 0 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.801+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 10 monclient(hunting): handle_auth_done global_id 5232 payload 274 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 1 -- 10.20.193.51:0/307256701 >> v1:10.20.193.125:6789/0 conn(0x7f09cc15c710 legacy=0x7f09cc06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 1 -- 10.20.193.51:0/307256701 >> v1:10.20.193.68:6789/0 conn(0x7f09cc150600 legacy=0x7f09cc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 1 monclient: found mon.noname-a 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.802+0000 7f09cbfff640 1 -- 10.20.193.51:0/307256701 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc15b520 con 0x7f09cc150220 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.803+0000 7f09cbfff640 1 --2- 10.20.193.51:0/307256701 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc150220 0x7f09cc0c73a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009970 tx=0x7f09b802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949a22fc5477b676 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 1 -- 10.20.193.51:0/307256701 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b803c070 con 0x7f09cc150220 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: dump: 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.814 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 1 -- 10.20.193.51:0/307256701 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b802fcd0 con 0x7f09cc150220 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 1 -- 10.20.193.51:0/307256701 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8037660 con 0x7f09cc150220 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09caffd640 10 monclient: dump: 2026-02-22T21:42:49.815 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 5 monclient: authenticate success, global_id 5232 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 4 set_mon_vals no callback set 2026-02-22T21:42:49.816 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 10 monclient: shutdown 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 -- 10.20.193.51:0/307256701 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc150220 msgr2=0x7f09cc0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 --2- 10.20.193.51:0/307256701 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc150220 0x7f09cc0c73a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009970 tx=0x7f09b802f000 comp rx=0 tx=0).stop 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 -- 10.20.193.51:0/307256701 shutdown_connections 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 --2- 10.20.193.51:0/307256701 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc150220 0x7f09cc0c73a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 -- 10.20.193.51:0/307256701 >> 10.20.193.51:0/307256701 conn(0x7f09cc0cd7b0 msgr2=0x7f09cc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 -- 10.20.193.51:0/307256701 shutdown_connections 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.805+0000 7f09d2a61640 1 -- 10.20.193.51:0/307256701 wait complete. 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient: build_initial_monmap 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient: monmap: 2026-02-22T21:42:49.817 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 librados: starting msgr at 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 librados: starting objecter 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 Processor -- start 2026-02-22T21:42:49.818 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 -- start start 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 librados: setting wanted keys 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 librados: calling monclient init 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient: init 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f09cc150220 0x7f09cc1f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f09cc150600 0x7f09cc149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc15c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:49.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0703b0 con 0x7f09cc15c710 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc15bae0 con 0x7f09cc150220 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc070530 con 0x7f09cc150600 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.806+0000 7f09d2a61640 10 monclient(hunting): authenticate will time out at 1445.386999s 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f09cc150220 0x7f09cc1f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35496/0 (socket says 10.20.193.51:35496) 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 1 -- 10.20.193.51:0/3865648095 learned_addr learned my addr 10.20.193.51:0/3865648095 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): get_auth_request con 0x7f09cc15c710 auth_method 0 2026-02-22T21:42:49.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cb7fe640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f09cc150600 0x7f09cc149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): get_auth_request con 0x7f09cc150220 auth_method 0 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09d0fd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cb7fe640 10 monclient(hunting): get_auth_request con 0x7f09cc150600 auth_method 0 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:49.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.807+0000 7f09cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 10 monclient(hunting): handle_auth_done global_id 5235 payload 824 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 -- 10.20.193.51:0/3865648095 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f09cc150600 msgr2=0x7f09cc149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f09cc150600 0x7f09cc149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 -- 10.20.193.51:0/3865648095 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f09cc150220 msgr2=0x7f09cc1f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f09cc150220 0x7f09cc1f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:49.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 10 monclient: _finish_hunting 0 2026-02-22T21:42:49.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 monclient: found mon.noname-a 2026-02-22T21:42:49.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:49.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.808+0000 7f09d0fd7640 1 -- 10.20.193.51:0/3865648095 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc0c9730 con 0x7f09cc15c710 2026-02-22T21:42:49.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.810+0000 7f09d0fd7640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09c000cc50 tx=0x7f09c000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.810+0000 7f09c8ff9640 10 client.?.objecter ms_handle_connect 0x7f09cc15c710 2026-02-22T21:42:49.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.810+0000 7f09c8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0003a70 con 0x7f09cc15c710 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: dump: 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.826 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c0003c10 con 0x7f09cc15c710 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 5 monclient: authenticate success, global_id 5235 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c00050d0 con 0x7f09cc15c710 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09ca7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:49.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09c8ff9640 10 monclient: dump: 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:49.828 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 10 monclient: _renew_subs 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc0c9a10 con 0x7f09cc15c710 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 10 client.5235.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 10 monclient: _renew_subs 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1e53f0 con 0x7f09cc15c710 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.811+0000 7f09d2a61640 1 librados: init done 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.812+0000 7f09d2a61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.812+0000 7f09d2a61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.812+0000 7f09d2a61640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.812+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc0c6410 con 0x7f09cc15c710 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.823+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f09c0005270 con 0x7f09cc15c710 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.824+0000 7f09c8ff9640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 0x7f099c07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.824+0000 7f09cbfff640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 0x7f099c07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.824+0000 7f09cbfff640 10 monclient: get_auth_request con 0x7f099c078e50 auth_method 0 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.824+0000 7f09c8ff9640 10 client.5235.objecter ms_handle_connect 0x7f099c078e50 2026-02-22T21:42:49.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.824+0000 7f09cbfff640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 0x7f099c07b310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09cc074940 tx=0x7f09b8039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.825+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f09c005e7a0 con 0x7f09cc15c710 2026-02-22T21:42:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.825+0000 7f09c8ff9640 10 client.5235.objecter ms_dispatch2 0x7f09cc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.825+0000 7f09c8ff9640 3 client.5235.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:49.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.825+0000 7f09c8ff9640 3 client.5235.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:49.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.825+0000 7f09c8ff9640 20 client.5235.objecter dump_active .. 0 homeless 2026-02-22T21:42:49.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.931+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f09cc0c6410 con 0x7f09cc15c710 2026-02-22T21:42:49.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.931+0000 7f09c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:49.933 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:49.931+0000 7f09c8ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:50.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.015+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "9", "target": ["mon-mgr", ""]}) -- 0x7f09cc0c73a0 con 0x7f099c078e50 2026-02-22T21:42:50.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.015+0000 7f09c8ff9640 1 -- 10.20.193.51:0/3865648095 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 9 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f09cc0c73a0 con 0x7f099c078e50 2026-02-22T21:42:50.016 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 9 to deep-scrub 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 10 librados: watch_flush enter 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 10 librados: watch_flush exit 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 20 client.5235.objecter shutdown clearing up homeless session... 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 10 client.5235.objecter successfully canceled tick 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 msgr2=0x7f099c07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 0x7f099c07b310 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09cc074940 tx=0x7f09b8039040 comp rx=0 tx=0).stop 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 10 monclient: shutdown 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 msgr2=0x7f09cc0c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f09c000cc50 tx=0x7f09c000a430 comp rx=0 tx=0).stop 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 shutdown_connections 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f099c078e50 0x7f099c07b310 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f09cc15c710 0x7f09cc0c8c80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f09cc150600 0x7f09cc149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 --2- 10.20.193.51:0/3865648095 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f09cc150220 0x7f09cc1f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 >> 10.20.193.51:0/3865648095 conn(0x7f09cc0cd7b0 msgr2=0x7f09cc14da40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 shutdown_connections 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 -- 10.20.193.51:0/3865648095 wait complete. 2026-02-22T21:42:50.018 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.016+0000 7f09d2a61640 1 librados: shutdown 2026-02-22T21:42:50.025 INFO:tasks.ceph:Scrubbing osd.13 2026-02-22T21:42:50.026 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:50.098 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.096+0000 7fb2030cd640 10 monclient: get_monmap_and_config 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.096+0000 7fb2030cd640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.096+0000 7fb2030cd640 10 monclient: monmap: 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:50.097564+0000 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:50.097564+0000 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.099 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 Processor -- start 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 -- start start 2026-02-22T21:42:50.100 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient: init 2026-02-22T21:42:50.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient: _add_conns ranks=[3,1,4,5,0,2] 2026-02-22T21:42:50.101 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb1fc153860 0x7fb1fc15af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-a con 0x7fb1fc153860 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb1fc0c9500 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb1fc0c7c50 0x7fb1fc0c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-b con 0x7fb1fc0c7c50 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0c8180 con 0x7fb1fc0c9500 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fb1fc0c8460 con 0x7fb1fc153860 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fb1fc0c8740 con 0x7fb1fc0c7c50 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.102 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.097+0000 7fb2030cd640 10 monclient(hunting): authenticate will time out at 1445.677947s 2026-02-22T21:42:50.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.111+0000 7fb200e42640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb1fc153860 0x7fb1fc15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fb1fc153860 0x7fb1fc15af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35506/0 (socket says 10.20.193.51:35506) 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 1 -- 10.20.193.51:0/2194647646 learned_addr learned my addr 10.20.193.51:0/2194647646 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 10 monclient(hunting): get_auth_request con 0x7fb1fc153860 auth_method 0 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.112+0000 7fb200e42640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.114+0000 7fb200e42640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.114+0000 7fb200e42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.114+0000 7fb200e42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 10 monclient(hunting): handle_auth_done global_id 5238 payload 274 2026-02-22T21:42:50.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 1 -- 10.20.193.51:0/2194647646 >> v2:10.20.193.68:3300/0 conn(0x7fb1fc0c7c50 msgr2=0x7fb1fc0c4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 1 --2- 10.20.193.51:0/2194647646 >> v2:10.20.193.68:3300/0 conn(0x7fb1fc0c7c50 0x7fb1fc0c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 1 -- 10.20.193.51:0/2194647646 >> v1:10.20.193.68:6789/0 conn(0x7fb1fc0c9500 legacy=0x7fb1fc15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 1 monclient: found mon.noname-a 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.115+0000 7fb200e42640 1 -- 10.20.193.51:0/2194647646 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc0c8a20 con 0x7fb1fc153860 2026-02-22T21:42:50.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.116+0000 7fb200e42640 1 --2- 10.20.193.51:0/2194647646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc153860 0x7fb1fc15af10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0c6660 tx=0x7fb1ec02eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00b05b967210a6c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 1 -- 10.20.193.51:0/2194647646 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec03c070 con 0x7fb1fc153860 2026-02-22T21:42:50.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:50.124 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: dump: 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.125 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 1 -- 10.20.193.51:0/2194647646 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1ec02fb40 con 0x7fb1fc153860 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 1 -- 10.20.193.51:0/2194647646 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec037440 con 0x7fb1fc153860 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb2030cd640 5 monclient: authenticate success, global_id 5238 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb2030cd640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb2030cd640 4 set_mon_vals no callback set 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.118+0000 7fb1f37fe640 10 monclient: dump: 2026-02-22T21:42:50.126 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 10 monclient: shutdown 2026-02-22T21:42:50.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 -- 10.20.193.51:0/2194647646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc153860 msgr2=0x7fb1fc15af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2194647646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc153860 0x7fb1fc15af10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0c6660 tx=0x7fb1ec02eed0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 -- 10.20.193.51:0/2194647646 shutdown_connections 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2194647646 >> v2:10.20.193.68:3300/0 conn(0x7fb1fc0c7c50 0x7fb1fc0c4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2194647646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc153860 0x7fb1fc15af10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 -- 10.20.193.51:0/2194647646 >> 10.20.193.51:0/2194647646 conn(0x7fb1fc0c9d50 msgr2=0x7fb1fc0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 -- 10.20.193.51:0/2194647646 shutdown_connections 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 1 -- 10.20.193.51:0/2194647646 wait complete. 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.119+0000 7fb2030cd640 10 monclient: monmap: 2026-02-22T21:42:50.128 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 librados: starting msgr at 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 librados: starting objecter 2026-02-22T21:42:50.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 Processor -- start 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 -- start start 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 librados: setting wanted keys 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 librados: calling monclient init 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient: init 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc0c7c50 0x7fb1fc1f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-a con 0x7fb1fc0c7c50 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb1fc153860 0x7fb1fc06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-b con 0x7fb1fc153860 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 0x7fb1fc141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): picked mon.noname-c con 0x7fb1fc06d7d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc15b450 con 0x7fb1fc0c7c50 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0c8fe0 con 0x7fb1fc153860 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc153c40 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb2030cd640 10 monclient(hunting): authenticate will time out at 1445.700572s 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb1f3fff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb1fc153860 0x7fb1fc06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.120+0000 7fb201643640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 0x7fb1fc141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 0x7fb1fc141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36862/0 (socket says 10.20.193.51:36862) 2026-02-22T21:42:50.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 1 -- 10.20.193.51:0/2419718643 learned_addr learned my addr 10.20.193.51:0/2419718643 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): get_auth_request con 0x7fb1fc06d7d0 auth_method 0 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): get_auth_request con 0x7fb1fc153860 auth_method 0 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb201643640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.121+0000 7fb1f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb200e42640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc0c7c50 0x7fb1fc1f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 10 monclient(hunting): handle_auth_done global_id 5237 payload 824 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 -- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb1fc153860 msgr2=0x7fb1fc06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb1fc153860 0x7fb1fc06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 -- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc0c7c50 msgr2=0x7fb1fc1f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb1fc0c7c50 0x7fb1fc1f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 monclient: found mon.noname-c 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.122+0000 7fb201643640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc14dbf0 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.124+0000 7fb201643640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 0x7fb1fc141510 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800b9e0 tx=0x7fb1f800beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.124+0000 7fb1f17fa640 10 client.?.objecter ms_handle_connect 0x7fb1fc06d7d0 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.124+0000 7fb1f17fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800c6b0 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: dump: 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.136 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1f8010070 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800cab0 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f17fa640 10 monclient: dump: 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.138 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb1f2ffd640 4 set_mon_vals no callback set 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb2030cd640 5 monclient: authenticate success, global_id 5237 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb2030cd640 10 monclient: _renew_subs 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb2030cd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.125+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc14ded0 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 client.5237.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 monclient: _renew_subs 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc14e490 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 1 librados: init done 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 client.5237.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 15 client.5237.objecter _session_command_op_assign -1 1 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 20 client.5237.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 client.5237.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.126+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1fc14eb50 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.137+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb1f801d030 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.137+0000 7fb1f17fa640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1cc078e50 0x7fb1cc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.137+0000 7fb200e42640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1cc078e50 0x7fb1cc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.138+0000 7fb200e42640 10 monclient: get_auth_request con 0x7fb1cc078e50 auth_method 0 2026-02-22T21:42:50.142 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.138+0000 7fb1f17fa640 10 client.5237.objecter ms_handle_connect 0x7fb1cc078e50 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.138+0000 7fb200e42640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1cc078e50 0x7fb1cc07b310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc15cf50 tx=0x7fb1ec033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb1f805d810 con 0x7fb1fc06d7d0 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 10 client.5237.objecter ms_dispatch2 0x7fb1fc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 3 client.5237.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 3 client.5237.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 10 client.5237.objecter checking command 1 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fb1cc081330 0x7fb1cc083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter _get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 15 client.5237.objecter _session_command_op_remove -1 1 2026-02-22T21:42:50.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter _get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 15 client.5237.objecter _session_command_op_assign 13 1 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 4 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 10 client.5237.objecter _send_command 1 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1f801be70 con 0x7fb1cc081330 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.139+0000 7fb1f17fa640 20 client.5237.objecter dump_active .. 0 homeless 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.140+0000 7fb1f3fff640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fb1cc081330 0x7fb1cc083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.140+0000 7fb1f3fff640 10 monclient: get_auth_request con 0x7fb1cc081330 auth_method 0 2026-02-22T21:42:50.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.141+0000 7fb1f3fff640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fb1cc081330 0x7fb1cc083750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.13 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.141+0000 7fb1f17fa640 10 client.5237.objecter ms_handle_connect 0x7fb1cc081330 2026-02-22T21:42:50.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.153+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== osd.13 v2:10.20.193.68:6800/762840177 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fb1f801be70 con 0x7fb1cc081330 2026-02-22T21:42:50.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.153+0000 7fb1f17fa640 10 client.5237.objecter ms_dispatch2 0x7fb1fc000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:50.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.153+0000 7fb1f17fa640 10 client.5237.objecter _finish_command 1 = system:0 2026-02-22T21:42:50.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.153+0000 7fb1f17fa640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 4 2026-02-22T21:42:50.155 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.153+0000 7fb1f17fa640 15 client.5237.objecter _session_command_op_remove 13 1 2026-02-22T21:42:50.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 10 client.5237.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:50.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 15 client.5237.objecter _session_command_op_assign -1 2 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 20 client.5237.objecter _get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 20 client.5237.objecter _get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 15 client.5237.objecter _session_command_op_remove -1 2 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 20 client.5237.objecter get_session s=0x7fb1cc080c60 osd=13 3 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 15 client.5237.objecter _session_command_op_assign 13 2 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 4 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 10 client.5237.objecter _send_command 2 2026-02-22T21:42:50.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.159+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 --> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb1fc15af10 con 0x7fb1cc081330 2026-02-22T21:42:50.174 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb1f17fa640 1 -- 10.20.193.51:0/2419718643 <== osd.13 v2:10.20.193.68:6800/762840177 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7fb1fc15af10 con 0x7fb1cc081330 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb1f17fa640 10 client.5237.objecter ms_dispatch2 0x7fb1fc000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb1f17fa640 10 client.5237.objecter _finish_command 2 = system:0 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb1f17fa640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 4 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb1f17fa640 15 client.5237.objecter _session_command_op_remove 13 2 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:50.175 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:50.176 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:50.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 10 librados: watch_flush enter 2026-02-22T21:42:50.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 10 librados: watch_flush exit 2026-02-22T21:42:50.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 10 client.5237.objecter close_session for osd.13 2026-02-22T21:42:50.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fb1cc081330 msgr2=0x7fb1cc083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.177 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.68:6800/762840177,v1:10.20.193.68:6801/762840177] conn(0x7fb1cc081330 0x7fb1cc083750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 20 client.5237.objecter put_session s=0x7fb1cc080c60 osd=13 1 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 20 client.5237.objecter shutdown clearing up homeless session... 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 10 client.5237.objecter successfully canceled tick 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1cc078e50 msgr2=0x7fb1cc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1cc078e50 0x7fb1cc07b310 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc15cf50 tx=0x7fb1ec033000 comp rx=0 tx=0).stop 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 10 monclient: shutdown 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 20 monclient: shutdown canceling and discarding version request 1 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 msgr2=0x7fb1fc141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 --2- 10.20.193.51:0/2419718643 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb1fc06d7d0 0x7fb1fc141510 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800b9e0 tx=0x7fb1f800beb0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.178 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb201643640 1 -- 10.20.193.51:0/2419718643 reap_dead start 2026-02-22T21:42:50.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 shutdown_connections 2026-02-22T21:42:50.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 >> 10.20.193.51:0/2419718643 conn(0x7fb1fc0c9d50 msgr2=0x7fb1fc0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 shutdown_connections 2026-02-22T21:42:50.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 -- 10.20.193.51:0/2419718643 wait complete. 2026-02-22T21:42:50.179 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.173+0000 7fb2030cd640 1 librados: shutdown 2026-02-22T21:42:50.183 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2026-02-22T21:42:50.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 10 monclient: get_monmap_and_config 2026-02-22T21:42:50.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 10 monclient: monmap: 2026-02-22T21:42:50.253 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:50.252068+0000 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:50.252068+0000 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:50.254 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 1 Processor -- start 2026-02-22T21:42:50.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 1 -- start start 2026-02-22T21:42:50.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.251+0000 7f68d0fc4640 10 monclient: init 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient: _add_conns ranks=[1,0,2,3,4,5] 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f68cc150220 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68cc15c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68cc0703b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc15a920 con 0x7f68cc15c710 2026-02-22T21:42:50.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc15af60 con 0x7f68cc150220 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc15b240 con 0x7f68cc0703b0 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68d0fc4640 10 monclient(hunting): authenticate will time out at 1445.832468s 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68c9d74640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f68cc15c710 0x7f68cc06fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58982/0 (socket says 10.20.193.51:58982) 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.252+0000 7f68c9d74640 1 -- 10.20.193.51:0/3088004523 learned_addr learned my addr 10.20.193.51:0/3088004523 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1701449525 0 0) 0x7f68cc15a920 con 0x7f68cc15c710 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _negotiate 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): my global_id is 5241 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68b8003620 con 0x7f68cc15c710 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791608159 0 0) 0x7f68cc15b240 con 0x7f68cc0703b0 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _negotiate 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): my global_id is 5240 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68cc15a920 con 0x7f68cc0703b0 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568825579 0 0) 0x7f68cc15af60 con 0x7f68cc150220 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _negotiate 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 10 monclient(hunting): my global_id is 5248 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.254+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68cc15b240 con 0x7f68cc150220 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.2 v1:10.20.193.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 725874876 0 0) 0x7f68cc15a920 con 0x7f68cc0703b0 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 >> v1:10.20.193.68:6789/0 conn(0x7f68cc150220 legacy=0x7f68cc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 >> v1:10.20.193.51:6789/0 conn(0x7f68cc15c710 legacy=0x7f68cc06fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 monclient: found mon.noname-c-legacy 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:10.20.193.125:6789/0 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> v1:10.20.193.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc15b520 con 0x7f68cc0703b0 2026-02-22T21:42:50.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.2 v1:10.20.193.125:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f68bc003260 con 0x7f68cc0703b0 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68d0fc4640 5 monclient: authenticate success, global_id 5240 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68d0fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: dump: 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.265 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 monclient: mon.c has (v2) addrs [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] but i'm connected to v1:10.20.193.125:6789/0, reconnecting 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 >> v1:10.20.193.125:6789/0 conn(0x7f68cc0703b0 legacy=0x7f68cc159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68b80038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): picked mon.a con 0x7f68b8005040 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68b8003de0 0x7f68b8004260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): picked mon.c con 0x7f68b8003de0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68b8009ab0 0x7f68b800bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): picked mon.b con 0x7f68b8009ab0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f68b800c490 con 0x7f68b8005040 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f68b800c640 con 0x7f68b8009ab0 2026-02-22T21:42:50.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f68b800c840 con 0x7f68b8003de0 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68c9573640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.255+0000 7f68d0fc4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68ca575640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68b80038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68ca575640 10 monclient(hunting): get_auth_request con 0x7f68b8005040 auth_method 0 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68ca575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68ca575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68ca575640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.256+0000 7f68cad76640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68b8009ab0 0x7f68b800bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68c9d74640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68b8003de0 0x7f68b8004260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): get_auth_request con 0x7f68b8009ab0 auth_method 0 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68c9d74640 10 monclient(hunting): get_auth_request con 0x7f68b8003de0 auth_method 0 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68c9d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68c9d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.257+0000 7f68c9d74640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 10 monclient(hunting): handle_auth_done global_id 5240 payload 293 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 -- 10.20.193.51:0/3088004523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68b8003de0 msgr2=0x7f68b8004260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68b8003de0 0x7f68b8004260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 -- 10.20.193.51:0/3088004523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68b8009ab0 msgr2=0x7f68b800bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68b8009ab0 0x7f68b800bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 monclient: found mon.a 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68cad76640 1 -- 10.20.193.51:0/3088004523 reap_dead start 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 -- 10.20.193.51:0/3088004523 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc15a920 con 0x7f68b8005040 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.258+0000 7f68ca575640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68b80038a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68c00031a0 tx=0x7f68c002ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717859d5e2c4367f server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0002dc0 con 0x7f68b8005040 2026-02-22T21:42:50.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.277 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: dump: 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.278 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68d0fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c00052d0 con 0x7f68b8005040 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 1 -- 10.20.193.51:0/3088004523 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c002f720 con 0x7f68b8005040 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68c9573640 10 monclient: dump: 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.279 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68d0fc4640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.260+0000 7f68d0fc4640 4 set_mon_vals no callback set 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 10 monclient: shutdown 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 msgr2=0x7f68b80038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68b80038a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68c00031a0 tx=0x7f68c002ef00 comp rx=0 tx=0).stop 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- 10.20.193.51:0/3088004523 shutdown_connections 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/3088004523 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68b80038a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- 10.20.193.51:0/3088004523 >> 10.20.193.51:0/3088004523 conn(0x7f68cc0cd7b0 msgr2=0x7f68cc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- 10.20.193.51:0/3088004523 shutdown_connections 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- 10.20.193.51:0/3088004523 wait complete. 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 10 monclient: monmap: 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:50.281 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 librados: starting msgr at 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 librados: starting objecter 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 Processor -- start 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 -- start start 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 librados: setting wanted keys 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 1 librados: calling monclient init 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.261+0000 7f68d0fc4640 10 monclient: init 2026-02-22T21:42:50.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f68b8005040 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68cc14dcb0 0x7f68cc153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc14dcb0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68cc1537b0 0x7f68cc14bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc1537b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc150600 con 0x7f68b8005040 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc15bae0 con 0x7f68cc1537b0 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc150780 con 0x7f68cc14dcb0 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68d0fc4640 10 monclient(hunting): authenticate will time out at 1445.842285s 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35544/0 (socket says 10.20.193.51:35544) 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 1 -- 10.20.193.51:0/1017630284 learned_addr learned my addr 10.20.193.51:0/1017630284 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): get_auth_request con 0x7f68b8005040 auth_method 0 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68cad76640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68cc1537b0 0x7f68cc14bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68c9d74640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68cc14dcb0 0x7f68cc153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.284 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68ca575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68cad76640 10 monclient(hunting): get_auth_request con 0x7f68cc1537b0 auth_method 0 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68cad76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68cad76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.262+0000 7f68cad76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): get_auth_request con 0x7f68cc14dcb0 auth_method 0 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68cad76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.263+0000 7f68c9d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 10 monclient(hunting): handle_auth_done global_id 5244 payload 824 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 -- 10.20.193.51:0/1017630284 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68cc14dcb0 msgr2=0x7f68cc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68cc14dcb0 0x7f68cc153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 -- 10.20.193.51:0/1017630284 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68cc1537b0 msgr2=0x7f68cc14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68cc1537b0 0x7f68cc14bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 monclient: found mon.noname-a 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.264+0000 7f68ca575640 1 -- 10.20.193.51:0/1017630284 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc1521d0 con 0x7f68b8005040 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.265+0000 7f68ca575640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68c00032d0 tx=0x7f68c0002ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.265+0000 7f68b2ffd640 10 client.?.objecter ms_handle_connect 0x7f68b8005040 2026-02-22T21:42:50.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.265+0000 7f68b2ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c002f720 con 0x7f68b8005040 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: dump: 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.287 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c0045070 con 0x7f68b8005040 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 5 monclient: authenticate success, global_id 5244 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 10 monclient: _renew_subs 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc1524b0 con 0x7f68b8005040 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c002fac0 con 0x7f68b8005040 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68b2ffd640 10 monclient: dump: 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.288 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68c8d72640 4 set_mon_vals no callback set 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 10 client.5244.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 10 monclient: _renew_subs 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc152a70 con 0x7f68b8005040 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.266+0000 7f68d0fc4640 1 librados: init done 2026-02-22T21:42:50.289 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.267+0000 7f68d0fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.267+0000 7f68d0fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.267+0000 7f68d0fc4640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.290 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.267+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc0c73a0 con 0x7f68b8005040 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.278+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f68c002fc60 con 0x7f68b8005040 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.279+0000 7f68b2ffd640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 0x7f689807b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.279+0000 7f68c9d74640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 0x7f689807b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.279+0000 7f68c9d74640 10 monclient: get_auth_request con 0x7f6898078e50 auth_method 0 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.279+0000 7f68b2ffd640 10 client.5244.objecter ms_handle_connect 0x7f6898078e50 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.279+0000 7f68c9d74640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 0x7f689807b310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f68b4002410 tx=0x7f68b4007ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.281+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f68c0005190 con 0x7f68b8005040 2026-02-22T21:42:50.291 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.281+0000 7f68b2ffd640 10 client.5244.objecter ms_dispatch2 0x7f68cc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:50.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.281+0000 7f68b2ffd640 3 client.5244.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:50.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.281+0000 7f68b2ffd640 3 client.5244.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:50.292 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.281+0000 7f68b2ffd640 20 client.5244.objecter dump_active .. 0 homeless 2026-02-22T21:42:50.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.380+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f68c0086dd0 con 0x7f68b8005040 2026-02-22T21:42:50.381 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.380+0000 7f68b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.380+0000 7f68b2ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:50.466 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.465+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "13", "target": ["mon-mgr", ""]}) -- 0x7f68cc074da0 con 0x7f6898078e50 2026-02-22T21:42:50.467 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.465+0000 7f68b2ffd640 1 -- 10.20.193.51:0/1017630284 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 13 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7f68cc074da0 con 0x7f6898078e50 2026-02-22T21:42:50.467 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 13 to deep-scrub 2026-02-22T21:42:50.468 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 10 librados: watch_flush enter 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 10 librados: watch_flush exit 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 20 client.5244.objecter shutdown clearing up homeless session... 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 10 client.5244.objecter successfully canceled tick 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 msgr2=0x7f689807b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 0x7f689807b310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f68b4002410 tx=0x7f68b4007ed0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 10 monclient: shutdown 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 msgr2=0x7f68cc1f21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.469 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68c00032d0 tx=0x7f68c0002ac0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 shutdown_connections 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6898078e50 0x7f689807b310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f68cc1537b0 0x7f68cc14bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f68cc14dcb0 0x7f68cc153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 --2- 10.20.193.51:0/1017630284 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f68b8005040 0x7f68cc1f21e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 >> 10.20.193.51:0/1017630284 conn(0x7f68cc0cd7b0 msgr2=0x7f68cc06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 shutdown_connections 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 -- 10.20.193.51:0/1017630284 wait complete. 2026-02-22T21:42:50.470 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.466+0000 7f68d0fc4640 1 librados: shutdown 2026-02-22T21:42:50.477 INFO:tasks.ceph:Scrubbing osd.2 2026-02-22T21:42:50.477 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 10 monclient: get_monmap_and_config 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 10 monclient: monmap: 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.547 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:50.545910+0000 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:50.545910+0000 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 1 Processor -- start 2026-02-22T21:42:50.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 1 -- start start 2026-02-22T21:42:50.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.545+0000 7f3b6a9d3640 10 monclient: init 2026-02-22T21:42:50.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient: _add_conns ranks=[4,3,2,0,5,1] 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3b64152910 0x7f3b640c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f3b64152910 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f3b64152cf0 0x7f3b64070810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3b64152cf0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b64159320 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b640c6390 con 0x7f3b64159320 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f3b640737a0 con 0x7f3b64152cf0 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f3b641597d0 con 0x7f3b64152910 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b6a9d3640 10 monclient(hunting): authenticate will time out at 1446.126238s 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b637fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f3b64152cf0 0x7f3b64070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b63fff640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f3b64152910 0x7f3b640c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b68f49640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f3b64159320 0x7f3b640c5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47630/0 (socket says 10.20.193.51:47630) 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b68f49640 1 -- 10.20.193.51:0/3856218613 learned_addr learned my addr 10.20.193.51:0/3856218613 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b63fff640 10 monclient(hunting): get_auth_request con 0x7f3b64152910 auth_method 0 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b63fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b63fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b637fe640 10 monclient(hunting): get_auth_request con 0x7f3b64152cf0 auth_method 0 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b637fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.546+0000 7f3b637fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.547+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 10 monclient(hunting): handle_auth_done global_id 5254 payload 274 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 1 -- 10.20.193.51:0/3856218613 >> v2:10.20.193.51:3300/0 conn(0x7f3b64152cf0 msgr2=0x7f3b64070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 1 --2- 10.20.193.51:0/3856218613 >> v2:10.20.193.51:3300/0 conn(0x7f3b64152cf0 0x7f3b64070810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 1 -- 10.20.193.51:0/3856218613 >> v1:10.20.193.125:6789/0 conn(0x7f3b64159320 legacy=0x7f3b640c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 1 monclient: found mon.noname-b 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.548+0000 7f3b63fff640 1 -- 10.20.193.51:0/3856218613 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b64073a60 con 0x7f3b64152910 2026-02-22T21:42:50.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.549+0000 7f3b63fff640 1 --2- 10.20.193.51:0/3856218613 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64152910 0x7f3b640c7c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3b64069a40 tx=0x7f3b5002ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f456bfa306bf9a server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 1 -- 10.20.193.51:0/3856218613 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5003c070 con 0x7f3b64152910 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: dump: 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.562 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 1 -- 10.20.193.51:0/3856218613 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5002faf0 con 0x7f3b64152910 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b6a9d3640 5 monclient: authenticate success, global_id 5254 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b6a9d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 1 -- 10.20.193.51:0/3856218613 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50037440 con 0x7f3b64152910 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b62ffd640 10 monclient: dump: 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.564 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b6a9d3640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.550+0000 7f3b6a9d3640 4 set_mon_vals no callback set 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 10 monclient: shutdown 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3856218613 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64152910 msgr2=0x7f3b640c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3856218613 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64152910 0x7f3b640c7c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3b64069a40 tx=0x7f3b5002ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3856218613 shutdown_connections 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3856218613 >> v2:10.20.193.51:3300/0 conn(0x7f3b64152cf0 0x7f3b64070810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3856218613 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64152910 0x7f3b640c7c50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3856218613 >> 10.20.193.51:0/3856218613 conn(0x7f3b640c9d50 msgr2=0x7f3b640ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.565 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3856218613 shutdown_connections 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3856218613 wait complete. 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.551+0000 7f3b6a9d3640 10 monclient: monmap: 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.566 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 librados: starting msgr at 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 librados: starting objecter 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 Processor -- start 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 -- start start 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 librados: setting wanted keys 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 librados: calling monclient init 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient: init 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.567 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3b64152910 0x7f3b641f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-c con 0x7f3b64152910 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 0x7f3b6413ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3b64152cf0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64159320 0x7f3b6413fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): picked mon.noname-b con 0x7f3b64159320 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64074300 con 0x7f3b64152cf0 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64074020 con 0x7f3b64159320 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f3b640745e0 con 0x7f3b64152910 2026-02-22T21:42:50.568 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b6a9d3640 10 monclient(hunting): authenticate will time out at 1446.132531s 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 0x7f3b6413ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b68f49640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64159320 0x7f3b6413fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b63fff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3b64152910 0x7f3b641f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 0x7f3b6413ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35550/0 (socket says 10.20.193.51:35550) 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 1 -- 10.20.193.51:0/3190101075 learned_addr learned my addr 10.20.193.51:0/3190101075 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 10 monclient(hunting): get_auth_request con 0x7f3b64152cf0 auth_method 0 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.552+0000 7f3b637fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): get_auth_request con 0x7f3b64159320 auth_method 0 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): get_auth_request con 0x7f3b64152910 auth_method 0 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.570 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b68f49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.553+0000 7f3b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 10 monclient(hunting): handle_auth_done global_id 5257 payload 824 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 -- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3b64152910 msgr2=0x7f3b641f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3b64152910 0x7f3b641f5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 -- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 msgr2=0x7f3b6413ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 0x7f3b6413ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.571 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 monclient: found mon.noname-b 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b68f49640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b641f1820 con 0x7f3b64159320 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.554+0000 7f3b637fe640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3b64152cf0 0x7f3b6413ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.555+0000 7f3b68f49640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64159320 0x7f3b6413fb20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800ca70 tx=0x7f3b5800cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.555+0000 7f3b60ff9640 10 client.?.objecter ms_handle_connect 0x7f3b64159320 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.555+0000 7f3b60ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58017070 con 0x7f3b64159320 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:50.572 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 10 monclient: dump: 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.573 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 5 monclient: authenticate success, global_id 5257 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 10 monclient: _renew_subs 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b641f1b00 con 0x7f3b64159320 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 10 client.5257.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 10 monclient: _renew_subs 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b64141a10 con 0x7f3b64159320 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.556+0000 7f3b6a9d3640 1 librados: init done 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 10 client.5257.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.574 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 15 client.5257.objecter _session_command_op_assign -1 1 2026-02-22T21:42:50.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 20 client.5257.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:50.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 10 client.5257.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.557+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b64074b00 con 0x7f3b64159320 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b64074b00 con 0x7f3b64159320 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b58021060 con 0x7f3b64159320 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b627fc640 4 set_mon_vals no callback set 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.568+0000 7f3b60ff9640 10 monclient: dump: 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.578 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f3b5801c440 con 0x7f3b64159320 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b60ff9640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3b34078e00 0x7f3b3407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b63fff640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3b34078e00 0x7f3b3407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.579 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b63fff640 10 monclient: get_auth_request con 0x7f3b34078e00 auth_method 0 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b63fff640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3b34078e00 0x7f3b3407b2c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3b50039000 tx=0x7f3b50033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.569+0000 7f3b60ff9640 10 client.5257.objecter ms_handle_connect 0x7f3b34078e00 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f3b5800add0 con 0x7f3b64159320 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 10 client.5257.objecter ms_dispatch2 0x7f3b64000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 3 client.5257.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 3 client.5257.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 10 client.5257.objecter checking command 1 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f3b34081340 0x7f3b34083760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter _get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 15 client.5257.objecter _session_command_op_remove -1 1 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter _get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.580 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 15 client.5257.objecter _session_command_op_assign 2 1 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 4 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 10 client.5257.objecter _send_command 1 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b58014ec0 con 0x7f3b34081340 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 20 client.5257.objecter dump_active .. 0 homeless 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3b5805dc40 con 0x7f3b64159320 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b60ff9640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.571+0000 7f3b637fe640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f3b34081340 0x7f3b34083760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.572+0000 7f3b637fe640 10 monclient: get_auth_request con 0x7f3b34081340 auth_method 0 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.573+0000 7f3b60ff9640 10 client.5257.objecter ms_handle_connect 0x7f3b34081340 2026-02-22T21:42:50.581 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.573+0000 7f3b637fe640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f3b34081340 0x7f3b34083760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.583+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== osd.2 v2:10.20.193.125:6808/1963822926 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f3b58014ec0 con 0x7f3b34081340 2026-02-22T21:42:50.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.583+0000 7f3b60ff9640 10 client.5257.objecter ms_dispatch2 0x7f3b64000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:50.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.583+0000 7f3b60ff9640 10 client.5257.objecter _finish_command 1 = system:0 2026-02-22T21:42:50.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.583+0000 7f3b60ff9640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 4 2026-02-22T21:42:50.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.583+0000 7f3b60ff9640 15 client.5257.objecter _session_command_op_remove 2 1 2026-02-22T21:42:50.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 10 client.5257.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:50.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 15 client.5257.objecter _session_command_op_assign -1 2 2026-02-22T21:42:50.590 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 20 client.5257.objecter _get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 20 client.5257.objecter _get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 15 client.5257.objecter _session_command_op_remove -1 2 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 20 client.5257.objecter get_session s=0x7f3b34080c70 osd=2 3 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 15 client.5257.objecter _session_command_op_assign 2 2 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 4 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 10 client.5257.objecter _send_command 2 2026-02-22T21:42:50.591 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.589+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 --> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3b641469f0 con 0x7f3b34081340 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b60ff9640 1 -- 10.20.193.51:0/3190101075 <== osd.2 v2:10.20.193.125:6808/1963822926 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3b641469f0 con 0x7f3b34081340 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b60ff9640 10 client.5257.objecter ms_dispatch2 0x7f3b64000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b60ff9640 10 client.5257.objecter _finish_command 2 = system:0 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b60ff9640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 4 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b60ff9640 15 client.5257.objecter _session_command_op_remove 2 2 2026-02-22T21:42:50.597 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:50.598 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:50.598 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:50.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 10 librados: watch_flush enter 2026-02-22T21:42:50.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 10 librados: watch_flush exit 2026-02-22T21:42:50.599 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 10 client.5257.objecter close_session for osd.2 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f3b34081340 msgr2=0x7f3b34083760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.125:6808/1963822926,v1:10.20.193.125:6809/1963822926] conn(0x7f3b34081340 0x7f3b34083760 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 20 client.5257.objecter put_session s=0x7f3b34080c70 osd=2 1 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 20 client.5257.objecter shutdown clearing up homeless session... 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 10 client.5257.objecter successfully canceled tick 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3b34078e00 msgr2=0x7f3b3407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3b34078e00 0x7f3b3407b2c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3b50039000 tx=0x7f3b50033000 comp rx=0 tx=0).stop 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 10 monclient: shutdown 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64159320 msgr2=0x7f3b6413fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 --2- 10.20.193.51:0/3190101075 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3b64159320 0x7f3b6413fb20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3b5800ca70 tx=0x7f3b5800cf40 comp rx=0 tx=0).stop 2026-02-22T21:42:50.600 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b68f49640 1 -- 10.20.193.51:0/3190101075 reap_dead start 2026-02-22T21:42:50.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 shutdown_connections 2026-02-22T21:42:50.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 >> 10.20.193.51:0/3190101075 conn(0x7f3b640c9d50 msgr2=0x7f3b64151bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.595+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 shutdown_connections 2026-02-22T21:42:50.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.596+0000 7f3b6a9d3640 1 -- 10.20.193.51:0/3190101075 wait complete. 2026-02-22T21:42:50.601 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.596+0000 7f3b6a9d3640 1 librados: shutdown 2026-02-22T21:42:50.604 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-22T21:42:50.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.671+0000 7f1056fb3640 10 monclient: get_monmap_and_config 2026-02-22T21:42:50.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.671+0000 7f1056fb3640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.673 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.671+0000 7f1056fb3640 10 monclient: monmap: 2026-02-22T21:42:50.673 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.673 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:50.672352+0000 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:50.672352+0000 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:50.674 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 Processor -- start 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 -- start start 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient: init 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient: _add_conns ranks=[0,2,4,3,1,5] 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1050150220 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f105015c710 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:50.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f10500703b0 0x7f10500c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-b con 0x7f10500703b0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105015a920 con 0x7f1050150220 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105015af60 con 0x7f105015c710 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f105015b240 con 0x7f10500703b0 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.672+0000 7f1056fb3640 10 monclient(hunting): authenticate will time out at 1446.252736s 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1054d28640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f1050150220 0x7f105015c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:58984/0 (socket says 10.20.193.51:58984) 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1054d28640 1 -- 10.20.193.51:0/243442193 learned_addr learned my addr 10.20.193.51:0/243442193 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:50.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 1 --2- 10.20.193.51:0/243442193 >> v2:10.20.193.68:3300/0 conn(0x7f10500703b0 0x7f10500c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): get_auth_request con 0x7f10500703b0 auth_method 0 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.673+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 1 -- 10.20.193.51:0/243442193 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793783695 0 0) 0x7f105015a920 con 0x7f1050150220 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 10 monclient(hunting): _negotiate 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 10 monclient(hunting): my global_id is 5253 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f10477fe640 1 -- 10.20.193.51:0/243442193 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f103c003620 con 0x7f1050150220 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 10 monclient(hunting): handle_auth_done global_id 5260 payload 274 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 1 -- 10.20.193.51:0/243442193 >> v1:10.20.193.125:6789/0 conn(0x7f105015c710 legacy=0x7f105006fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 1 -- 10.20.193.51:0/243442193 >> v1:10.20.193.51:6789/0 conn(0x7f1050150220 legacy=0x7f105015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 1 monclient: found mon.noname-b 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.674+0000 7f1055529640 1 -- 10.20.193.51:0/243442193 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105015b520 con 0x7f10500703b0 2026-02-22T21:42:50.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.675+0000 7f1055529640 1 --2- 10.20.193.51:0/243442193 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c73a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f104c009dc0 tx=0x7f104c02f2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb23792e7f80f72 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 1 -- 10.20.193.51:0/243442193 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c03c070 con 0x7f10500703b0 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: dump: 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.688 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 1 -- 10.20.193.51:0/243442193 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c037440 con 0x7f10500703b0 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 5 monclient: authenticate success, global_id 5260 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 1 -- 10.20.193.51:0/243442193 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c037810 con 0x7f10500703b0 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 4 set_mon_vals no callback set 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f10477fe640 10 monclient: dump: 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.690 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 10 monclient: shutdown 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 -- 10.20.193.51:0/243442193 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 msgr2=0x7f10500c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 --2- 10.20.193.51:0/243442193 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c73a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f104c009dc0 tx=0x7f104c02f2f0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 -- 10.20.193.51:0/243442193 shutdown_connections 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 --2- 10.20.193.51:0/243442193 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c73a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 -- 10.20.193.51:0/243442193 >> 10.20.193.51:0/243442193 conn(0x7f10500cd7b0 msgr2=0x7f10500cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 -- 10.20.193.51:0/243442193 shutdown_connections 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.677+0000 7f1056fb3640 1 -- 10.20.193.51:0/243442193 wait complete. 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient: monmap: 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:50.692 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 librados: starting msgr at 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 librados: starting objecter 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 Processor -- start 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 -- start start 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 librados: setting wanted keys 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 librados: calling monclient init 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient: init 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.693 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-b con 0x7f10500703b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 0x7f10501f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-a con 0x7f1050150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 0x7f10500be810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): picked mon.noname-c con 0x7f105015c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f1050150600 con 0x7f1050150220 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f105015bae0 con 0x7f10500703b0 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f1050150780 con 0x7f105015c710 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.678+0000 7f1056fb3640 10 monclient(hunting): authenticate will time out at 1446.258993s 2026-02-22T21:42:50.694 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 0x7f10501f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 0x7f10501f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35552/0 (socket says 10.20.193.51:35552) 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 1 -- 10.20.193.51:0/357383844 learned_addr learned my addr 10.20.193.51:0/357383844 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): get_auth_request con 0x7f1050150220 auth_method 0 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 0x7f10500be810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): get_auth_request con 0x7f10500703b0 auth_method 0 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): get_auth_request con 0x7f105015c710 auth_method 0 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1047fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1054d28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.679+0000 7f1055529640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 10 monclient(hunting): handle_auth_done global_id 5255 payload 824 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 -- 10.20.193.51:0/357383844 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 msgr2=0x7f10500c1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 -- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 msgr2=0x7f10501f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 0x7f10501f7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 monclient: found mon.noname-c 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.681+0000 7f1055529640 1 -- 10.20.193.51:0/357383844 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1050073160 con 0x7f105015c710 2026-02-22T21:42:50.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.682+0000 7f1055529640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 0x7f10500be810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f104c036000 tx=0x7f104c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.682+0000 7f10457fa640 10 client.?.objecter ms_handle_connect 0x7f105015c710 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.682+0000 7f10457fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c03c050 con 0x7f105015c710 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 10 monclient: dump: 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.698 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f10457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 5 monclient: authenticate success, global_id 5255 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _renew_subs 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1050073440 con 0x7f105015c710 2026-02-22T21:42:50.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 client.5255.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _renew_subs 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1050073a00 con 0x7f105015c710 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 1 librados: init done 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 10 monclient: _send_mon_message to mon.c at v2:10.20.193.125:3300/0 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.683+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10500c73a0 con 0x7f105015c710 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c048070 con 0x7f105015c710 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c041a10 con 0x7f105015c710 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f10457fa640 10 monclient: dump: 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.701 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.702 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.684+0000 7f1046ffd640 4 set_mon_vals no callback set 2026-02-22T21:42:50.791 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.789+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f104c041bb0 con 0x7f105015c710 2026-02-22T21:42:50.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.790+0000 7f10457fa640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 0x7f102007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.790+0000 7f1054d28640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 0x7f102007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.790+0000 7f1054d28640 10 monclient: get_auth_request con 0x7f1020078e00 auth_method 0 2026-02-22T21:42:50.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.790+0000 7f1054d28640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 0x7f102007b2c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1040005980 tx=0x7f1040007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.792 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.790+0000 7f10457fa640 10 client.5255.objecter ms_handle_connect 0x7f1020078e00 2026-02-22T21:42:50.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.791+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f104c084340 con 0x7f105015c710 2026-02-22T21:42:50.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.791+0000 7f10457fa640 10 client.5255.objecter ms_dispatch2 0x7f1050000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:50.793 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.791+0000 7f10457fa640 3 client.5255.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:50.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.791+0000 7f10457fa640 3 client.5255.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:50.794 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.791+0000 7f10457fa640 20 client.5255.objecter dump_active .. 0 homeless 2026-02-22T21:42:50.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.797+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mon.2 v2:10.20.193.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f104c088300 con 0x7f105015c710 2026-02-22T21:42:50.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.797+0000 7f10457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.799 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.797+0000 7f10457fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:50.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.879+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f1050074da0 con 0x7f1020078e00 2026-02-22T21:42:50.881 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.879+0000 7f10457fa640 1 -- 10.20.193.51:0/357383844 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1050074da0 con 0x7f1020078e00 2026-02-22T21:42:50.881 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T21:42:50.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.879+0000 7f1056fb3640 10 librados: watch_flush enter 2026-02-22T21:42:50.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 10 librados: watch_flush exit 2026-02-22T21:42:50.883 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 20 client.5255.objecter shutdown clearing up homeless session... 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 10 client.5255.objecter successfully canceled tick 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 msgr2=0x7f102007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 0x7f102007b2c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1040005980 tx=0x7f1040007dd0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 10 monclient: shutdown 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 msgr2=0x7f10500be810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 0x7f10500be810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f104c036000 tx=0x7f104c004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 shutdown_connections 2026-02-22T21:42:50.884 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f1020078e00 0x7f102007b2c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f105015c710 0x7f10500be810 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f1050150220 0x7f10501f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 --2- 10.20.193.51:0/357383844 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f10500703b0 0x7f10500c1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 >> 10.20.193.51:0/357383844 conn(0x7f10500cd7b0 msgr2=0x7f10500cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 shutdown_connections 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 -- 10.20.193.51:0/357383844 wait complete. 2026-02-22T21:42:50.885 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.880+0000 7f1056fb3640 1 librados: shutdown 2026-02-22T21:42:50.889 INFO:tasks.ceph:Scrubbing osd.6 2026-02-22T21:42:50.890 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:50.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.958+0000 7fbe95996640 10 monclient: get_monmap_and_config 2026-02-22T21:42:50.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.958+0000 7fbe95996640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.958+0000 7fbe95996640 10 monclient: monmap: 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:50.959242+0000 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:50.959242+0000 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:50.961 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.958+0000 7fbe95996640 1 Processor -- start 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 -- start start 2026-02-22T21:42:50.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient: init 2026-02-22T21:42:50.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient: _add_conns ranks=[1,4,3,5,2,0] 2026-02-22T21:42:50.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fbe90155c40 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbe90156020 0x7fbe90073440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90156020 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe90073980 0x7fbe9015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-a con 0x7fbe90073980 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe900c8360 con 0x7fbe90155c40 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fbe90073df0 con 0x7fbe90073980 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fbe900c8620 con 0x7fbe90156020 2026-02-22T21:42:50.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.959+0000 7fbe95996640 10 monclient(hunting): authenticate will time out at 1446.539591s 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fbe90156020 0x7fbe90073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe90073980 0x7fbe9015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbe90073980 0x7fbe9015d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35560/0 (socket says 10.20.193.51:35560) 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 1 -- 10.20.193.51:0/821945485 learned_addr learned my addr 10.20.193.51:0/821945485 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): get_auth_request con 0x7fbe90073980 auth_method 0 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe90156020 auth_method 0 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.960+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_done global_id 5259 payload 274 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 1 -- 10.20.193.51:0/821945485 >> v2:10.20.193.68:3300/0 conn(0x7fbe90156020 msgr2=0x7fbe90073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:50.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 1 --2- 10.20.193.51:0/821945485 >> v2:10.20.193.68:3300/0 conn(0x7fbe90156020 0x7fbe90073440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 1 -- 10.20.193.51:0/821945485 >> v1:10.20.193.68:6789/0 conn(0x7fbe90155c40 legacy=0x7fbe900c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.961+0000 7fbe8f7fe640 1 -- 10.20.193.51:0/821945485 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900c8900 con 0x7fbe90073980 2026-02-22T21:42:50.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.962+0000 7fbe8f7fe640 1 --2- 10.20.193.51:0/821945485 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe9015d1d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbe900c9a50 tx=0x7fbe8002f270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ab790a1884b6d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 1 -- 10.20.193.51:0/821945485 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8002fd70 con 0x7fbe90073980 2026-02-22T21:42:50.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: dump: 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.977 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 5 monclient: authenticate success, global_id 5259 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 1 -- 10.20.193.51:0/821945485 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe800373f0 con 0x7fbe90073980 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 1 -- 10.20.193.51:0/821945485 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe800377c0 con 0x7fbe90073980 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe8dffb640 10 monclient: dump: 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.979 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 4 set_mon_vals no callback set 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 10 monclient: shutdown 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 -- 10.20.193.51:0/821945485 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 msgr2=0x7fbe9015d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 --2- 10.20.193.51:0/821945485 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe9015d1d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbe900c9a50 tx=0x7fbe8002f270 comp rx=0 tx=0).stop 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 -- 10.20.193.51:0/821945485 shutdown_connections 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 --2- 10.20.193.51:0/821945485 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe9015d1d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 --2- 10.20.193.51:0/821945485 >> v2:10.20.193.68:3300/0 conn(0x7fbe90156020 0x7fbe90073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 -- 10.20.193.51:0/821945485 >> 10.20.193.51:0/821945485 conn(0x7fbe900c9d50 msgr2=0x7fbe900ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 -- 10.20.193.51:0/821945485 shutdown_connections 2026-02-22T21:42:50.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.964+0000 7fbe95996640 1 -- 10.20.193.51:0/821945485 wait complete. 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 10 monclient: build_initial_monmap 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 10 monclient: monmap: 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:50.981 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 librados: starting msgr at 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 librados: starting objecter 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 Processor -- start 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 -- start start 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 librados: setting wanted keys 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 1 librados: calling monclient init 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.965+0000 7fbe95996640 10 monclient: init 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe901f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.982 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-a con 0x7fbe90073980 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe90155c40 0x7fbe901570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90155c40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe90156020 0x7fbe90154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): picked mon.noname-c con 0x7fbe90156020 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900c91a0 con 0x7fbe90073980 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900c8ec0 con 0x7fbe90155c40 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900c9480 con 0x7fbe90156020 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe95996640 10 monclient(hunting): authenticate will time out at 1446.546214s 2026-02-22T21:42:50.983 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe901f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe901f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35568/0 (socket says 10.20.193.51:35568) 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 1 -- 10.20.193.51:0/3144183708 learned_addr learned my addr 10.20.193.51:0/3144183708 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): get_auth_request con 0x7fbe90073980 auth_method 0 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8f7fe640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe90156020 0x7fbe90154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8e7fc640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe90155c40 0x7fbe901570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.984 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe90155c40 auth_method 0 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8e7fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8e7fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8f7fe640 10 monclient(hunting): get_auth_request con 0x7fbe90156020 auth_method 0 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.966+0000 7fbe8f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:50.985 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.967+0000 7fbe8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 10 monclient(hunting): handle_auth_done global_id 5262 payload 824 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 -- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe90156020 msgr2=0x7fbe90154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbe90156020 0x7fbe90154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 -- 10.20.193.51:0/3144183708 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe90155c40 msgr2=0x7fbe901570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbe90155c40 0x7fbe901570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:50.986 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 monclient: found mon.noname-a 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.968+0000 7fbe8effd640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe901f3b50 con 0x7fbe90073980 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.969+0000 7fbe8effd640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe901f3610 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c580 tx=0x7fbe8400ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.969+0000 7fbe67fff640 10 client.?.objecter ms_handle_connect 0x7fbe90073980 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.969+0000 7fbe67fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84017070 con 0x7fbe90073980 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: dump: 2026-02-22T21:42:50.987 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:50.988 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 5 monclient: authenticate success, global_id 5262 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe840052d0 con 0x7fbe90073980 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 10 monclient: _renew_subs 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901f3e20 con 0x7fbe90073980 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe840036b0 con 0x7fbe90073980 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe67fff640 10 monclient: dump: 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:50.989 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe8d7fa640 4 set_mon_vals no callback set 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 10 client.5262.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 10 monclient: _renew_subs 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe901565e0 con 0x7fbe90073980 2026-02-22T21:42:50.990 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.970+0000 7fbe95996640 1 librados: init done 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 10 client.5262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 15 client.5262.objecter _session_command_op_assign -1 1 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 20 client.5262.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 10 client.5262.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:50.991 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.971+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe90073440 con 0x7fbe90073980 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.982+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fbe84003850 con 0x7fbe90073980 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.983+0000 7fbe67fff640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe60078e00 0x7fbe6007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.983+0000 7fbe8e7fc640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe60078e00 0x7fbe6007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.983+0000 7fbe8e7fc640 10 monclient: get_auth_request con 0x7fbe60078e00 auth_method 0 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.983+0000 7fbe8e7fc640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe60078e00 0x7fbe6007b2c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009c40 tx=0x7fbe78009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.983+0000 7fbe67fff640 10 client.5262.objecter ms_handle_connect 0x7fbe60078e00 2026-02-22T21:42:50.993 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fbe84096cc0 con 0x7fbe90073980 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 10 client.5262.objecter ms_dispatch2 0x7fbe90000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 3 client.5262.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 3 client.5262.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 10 client.5262.objecter checking command 1 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fbe600812e0 0x7fbe60083700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter _get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 15 client.5262.objecter _session_command_op_remove -1 1 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter _get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 15 client.5262.objecter _session_command_op_assign 6 1 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 4 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 10 client.5262.objecter _send_command 1 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe60083dd0 con 0x7fbe600812e0 2026-02-22T21:42:50.994 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 20 client.5262.objecter dump_active .. 0 homeless 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbe8405d780 con 0x7fbe90073980 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe67fff640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe8f7fe640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fbe600812e0 0x7fbe60083700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.985+0000 7fbe8f7fe640 10 monclient: get_auth_request con 0x7fbe600812e0 auth_method 0 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.987+0000 7fbe8f7fe640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fbe600812e0 0x7fbe60083700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:50.995 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.987+0000 7fbe67fff640 10 client.5262.objecter ms_handle_connect 0x7fbe600812e0 2026-02-22T21:42:50.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.997+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== osd.6 v2:10.20.193.125:6824/1184344866 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fbe60083dd0 con 0x7fbe600812e0 2026-02-22T21:42:50.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.997+0000 7fbe67fff640 10 client.5262.objecter ms_dispatch2 0x7fbe90000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:50.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.997+0000 7fbe67fff640 10 client.5262.objecter _finish_command 1 = system:0 2026-02-22T21:42:50.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.997+0000 7fbe67fff640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 4 2026-02-22T21:42:50.999 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:50.997+0000 7fbe67fff640 15 client.5262.objecter _session_command_op_remove 6 1 2026-02-22T21:42:51.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 10 client.5262.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:51.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 15 client.5262.objecter _session_command_op_assign -1 2 2026-02-22T21:42:51.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 20 client.5262.objecter _get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:51.005 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 20 client.5262.objecter _get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 15 client.5262.objecter _session_command_op_remove -1 2 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 20 client.5262.objecter get_session s=0x7fbe60080c10 osd=6 3 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 15 client.5262.objecter _session_command_op_assign 6 2 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 4 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 10 client.5262.objecter _send_command 2 2026-02-22T21:42:51.006 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.003+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 --> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbe90152640 con 0x7fbe600812e0 2026-02-22T21:42:51.012 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.010+0000 7fbe67fff640 1 -- 10.20.193.51:0/3144183708 <== osd.6 v2:10.20.193.125:6824/1184344866 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbe90152640 con 0x7fbe600812e0 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.010+0000 7fbe67fff640 10 client.5262.objecter ms_dispatch2 0x7fbe90000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.010+0000 7fbe67fff640 10 client.5262.objecter _finish_command 2 = system:0 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.010+0000 7fbe67fff640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 4 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.010+0000 7fbe67fff640 15 client.5262.objecter _session_command_op_remove 6 2 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:51.013 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 10 librados: watch_flush enter 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 10 librados: watch_flush exit 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 10 client.5262.objecter close_session for osd.6 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fbe600812e0 msgr2=0x7fbe60083700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.125:6824/1184344866,v1:10.20.193.125:6825/1184344866] conn(0x7fbe600812e0 0x7fbe60083700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 20 client.5262.objecter put_session s=0x7fbe60080c10 osd=6 1 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 20 client.5262.objecter shutdown clearing up homeless session... 2026-02-22T21:42:51.015 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 10 client.5262.objecter successfully canceled tick 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe60078e00 msgr2=0x7fbe6007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbe60078e00 0x7fbe6007b2c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009c40 tx=0x7fbe78009290 comp rx=0 tx=0).stop 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 10 monclient: shutdown 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 msgr2=0x7fbe901f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 --2- 10.20.193.51:0/3144183708 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbe90073980 0x7fbe901f3610 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c580 tx=0x7fbe8400ca50 comp rx=0 tx=0).stop 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe8f7fe640 1 -- 10.20.193.51:0/3144183708 reap_dead start 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 shutdown_connections 2026-02-22T21:42:51.016 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 >> 10.20.193.51:0/3144183708 conn(0x7fbe900c9d50 msgr2=0x7fbe9015d040 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 shutdown_connections 2026-02-22T21:42:51.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 -- 10.20.193.51:0/3144183708 wait complete. 2026-02-22T21:42:51.017 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.011+0000 7fbe95996640 1 librados: shutdown 2026-02-22T21:42:51.021 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2026-02-22T21:42:51.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 10 monclient: get_monmap_and_config 2026-02-22T21:42:51.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.089 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 10 monclient: monmap: 2026-02-22T21:42:51.089 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:51.088033+0000 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:51.088033+0000 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:51.090 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:51.091 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 1 Processor -- start 2026-02-22T21:42:51.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 1 -- start start 2026-02-22T21:42:51.091 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.087+0000 7fa0f4858640 10 monclient: init 2026-02-22T21:42:51.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient: _add_conns ranks=[0,3,1,2,5,4] 2026-02-22T21:42:51.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0ec155fb0 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:51.093 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa0ec15da30 0x7fa0ec15c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ec15da30 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fa0ec0ccc00 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ec0d05b0 con 0x7fa0ec155fb0 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ec06f650 con 0x7fa0ec0ccc00 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fa0ec0709d0 con 0x7fa0ec15da30 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f4858640 10 monclient(hunting): authenticate will time out at 1446.668434s 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f1dcc640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fa0ec15da30 0x7fa0ec15c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f2dce640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fa0ec0ccc00 0x7fa0ec0caff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:39030/0 (socket says 10.20.193.51:39030) 2026-02-22T21:42:51.094 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.088+0000 7fa0f2dce640 1 -- 10.20.193.51:0/2832888236 learned_addr learned my addr 10.20.193.51:0/2832888236 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): get_auth_request con 0x7fa0ec15da30 auth_method 0 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.089+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 1 -- 10.20.193.51:0/2832888236 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330210745 0 0) 0x7fa0ec06f650 con 0x7fa0ec0ccc00 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.095 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 10 monclient(hunting): my global_id is 5275 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f15cb640 1 -- 10.20.193.51:0/2832888236 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0dc003620 con 0x7fa0ec0ccc00 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_done global_id 5265 payload 274 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 1 -- 10.20.193.51:0/2832888236 >> v1:10.20.193.68:6789/0 conn(0x7fa0ec0ccc00 legacy=0x7fa0ec0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 1 -- 10.20.193.51:0/2832888236 >> v1:10.20.193.51:6789/0 conn(0x7fa0ec155fb0 legacy=0x7fa0ec15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.096 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.090+0000 7fa0f1dcc640 1 -- 10.20.193.51:0/2832888236 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec06f930 con 0x7fa0ec15da30 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.092+0000 7fa0f1dcc640 1 --2- 10.20.193.51:0/2832888236 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec15da30 0x7fa0ec15c8b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80098a0 tx=0x7fa0d802ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d89e05f77ae16d5 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.094+0000 7fa0f15cb640 1 -- 10.20.193.51:0/2832888236 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d803c070 con 0x7fa0ec15da30 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.094+0000 7fa0f15cb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.094+0000 7fa0f15cb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.094+0000 7fa0f15cb640 10 monclient: dump: 2026-02-22T21:42:51.104 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.105 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 1 -- 10.20.193.51:0/2832888236 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d802fca0 con 0x7fa0ec15da30 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 5 monclient: authenticate success, global_id 5265 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 1 -- 10.20.193.51:0/2832888236 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8037660 con 0x7fa0ec15da30 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 4 set_mon_vals no callback set 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f15cb640 10 monclient: dump: 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.106 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 10 monclient: shutdown 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.107 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 -- 10.20.193.51:0/2832888236 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec15da30 msgr2=0x7fa0ec15c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2832888236 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec15da30 0x7fa0ec15c8b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80098a0 tx=0x7fa0d802ef90 comp rx=0 tx=0).stop 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 -- 10.20.193.51:0/2832888236 shutdown_connections 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2832888236 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec15da30 0x7fa0ec15c8b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 -- 10.20.193.51:0/2832888236 >> 10.20.193.51:0/2832888236 conn(0x7fa0ec0cc0e0 msgr2=0x7fa0ec0cc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 -- 10.20.193.51:0/2832888236 shutdown_connections 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 1 -- 10.20.193.51:0/2832888236 wait complete. 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.095+0000 7fa0f4858640 10 monclient: monmap: 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:51.108 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 librados: starting msgr at 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 librados: starting objecter 2026-02-22T21:42:51.109 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 Processor -- start 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 -- start start 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 librados: setting wanted keys 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 librados: calling monclient init 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient: init 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa0ec0ccc00 0x7fa0ec0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-c con 0x7fa0ec0ccc00 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa0ec155fb0 0x7fa0ec1f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.110 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-b con 0x7fa0ec155fb0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ec145eb0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec06fef0 con 0x7fa0ec145eb0 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec070540 con 0x7fa0ec155fb0 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec0701d0 con 0x7fa0ec0ccc00 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.096+0000 7fa0f4858640 10 monclient(hunting): authenticate will time out at 1446.676979s 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.111 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35580/0 (socket says 10.20.193.51:35580) 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 1 -- 10.20.193.51:0/2230126041 learned_addr learned my addr 10.20.193.51:0/2230126041 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): get_auth_request con 0x7fa0ec145eb0 auth_method 0 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f1dcc640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa0ec155fb0 0x7fa0ec1f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa0ec0ccc00 0x7fa0ec0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f2dce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.112 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f1dcc640 10 monclient(hunting): get_auth_request con 0x7fa0ec155fb0 auth_method 0 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f1dcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f1dcc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f1dcc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): get_auth_request con 0x7fa0ec0ccc00 auth_method 0 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.097+0000 7fa0f25cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f1dcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 10 monclient(hunting): handle_auth_done global_id 5271 payload 824 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 -- 10.20.193.51:0/2230126041 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa0ec0ccc00 msgr2=0x7fa0ec0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa0ec0ccc00 0x7fa0ec0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 -- 10.20.193.51:0/2230126041 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa0ec155fb0 msgr2=0x7fa0ec1f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa0ec155fb0 0x7fa0ec1f4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.114 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.098+0000 7fa0f2dce640 1 -- 10.20.193.51:0/2230126041 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec0cdd80 con 0x7fa0ec145eb0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.100+0000 7fa0f2dce640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000cb40 tx=0x7fa0e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.100+0000 7fa0d6ffd640 10 client.?.objecter ms_handle_connect 0x7fa0ec145eb0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.100+0000 7fa0d6ffd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000ad50 con 0x7fa0ec145eb0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: dump: 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.115 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e0012400 con 0x7fa0ec145eb0 2026-02-22T21:42:51.116 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00127f0 con 0x7fa0ec145eb0 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0d6ffd640 10 monclient: dump: 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.117 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f0dca640 4 set_mon_vals no callback set 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 5 monclient: authenticate success, global_id 5271 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 10 monclient: _renew_subs 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0ec0ce060 con 0x7fa0ec145eb0 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 10 client.5271.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 10 monclient: _renew_subs 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0ec0ce620 con 0x7fa0ec145eb0 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.101+0000 7fa0f4858640 1 librados: init done 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.102+0000 7fa0f4858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.102+0000 7fa0f4858640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.102+0000 7fa0f4858640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.102+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0ec069170 con 0x7fa0ec145eb0 2026-02-22T21:42:51.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.113+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fa0e0012990 con 0x7fa0ec145eb0 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.114+0000 7fa0d6ffd640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 0x7fa0bc07b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.114+0000 7fa0f25cd640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 0x7fa0bc07b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.114+0000 7fa0f25cd640 10 monclient: get_auth_request con 0x7fa0bc078e50 auth_method 0 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.114+0000 7fa0d6ffd640 10 client.5271.objecter ms_handle_connect 0x7fa0bc078e50 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.114+0000 7fa0f25cd640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 0x7fa0bc07b310 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4002410 tx=0x7fa0e4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.115+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fa0e005e930 con 0x7fa0ec145eb0 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.115+0000 7fa0d6ffd640 10 client.5271.objecter ms_dispatch2 0x7fa0ec000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:51.122 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.115+0000 7fa0d6ffd640 3 client.5271.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:51.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.115+0000 7fa0d6ffd640 3 client.5271.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:51.123 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.115+0000 7fa0d6ffd640 20 client.5271.objecter dump_active .. 0 homeless 2026-02-22T21:42:51.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.217+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fa0e00628f0 con 0x7fa0ec145eb0 2026-02-22T21:42:51.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.217+0000 7fa0d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.218 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.217+0000 7fa0d6ffd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:51.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.299+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fa0ec15d750 con 0x7fa0bc078e50 2026-02-22T21:42:51.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.299+0000 7fa0d6ffd640 1 -- 10.20.193.51:0/2230126041 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa0ec15d750 con 0x7fa0bc078e50 2026-02-22T21:42:51.301 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 6 to deep-scrub 2026-02-22T21:42:51.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 10 librados: watch_flush enter 2026-02-22T21:42:51.303 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 10 librados: watch_flush exit 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 20 client.5271.objecter shutdown clearing up homeless session... 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 10 client.5271.objecter successfully canceled tick 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 msgr2=0x7fa0bc07b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 0x7fa0bc07b310 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4002410 tx=0x7fa0e4007ef0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 10 monclient: shutdown 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 msgr2=0x7fa0ec0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000cb40 tx=0x7fa0e000a430 comp rx=0 tx=0).stop 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 shutdown_connections 2026-02-22T21:42:51.304 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fa0bc078e50 0x7fa0bc07b310 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fa0ec145eb0 0x7fa0ec0cd440 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fa0ec155fb0 0x7fa0ec1f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 --2- 10.20.193.51:0/2230126041 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fa0ec0ccc00 0x7fa0ec0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 >> 10.20.193.51:0/2230126041 conn(0x7fa0ec0cc0e0 msgr2=0x7fa0ec15fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 shutdown_connections 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 -- 10.20.193.51:0/2230126041 wait complete. 2026-02-22T21:42:51.305 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.300+0000 7fa0f4858640 1 librados: shutdown 2026-02-22T21:42:51.310 INFO:tasks.ceph:Scrubbing osd.10 2026-02-22T21:42:51.311 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 10 monclient: get_monmap_and_config 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 10 monclient: monmap: 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:51.378794+0000 2026-02-22T21:42:51.380 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:51.378794+0000 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:51.381 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 1 Processor -- start 2026-02-22T21:42:51.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 1 -- start start 2026-02-22T21:42:51.382 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.378+0000 7f12dbe58640 10 monclient: init 2026-02-22T21:42:51.390 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient: _add_conns ranks=[4,5,0,3,1,2] 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f12d4153860 0x7f12d415af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-b con 0x7f12d4153860 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f12d40c9500 0x7f12d415d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-c con 0x7f12d40c9500 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12d40c7c50 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d40c8180 con 0x7f12d40c7c50 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f12d40c8460 con 0x7f12d4153860 2026-02-22T21:42:51.391 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f12d40c8740 con 0x7f12d40c9500 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12dbe58640 10 monclient(hunting): authenticate will time out at 1446.959128s 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12da3ce640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f12d40c7c50 0x7f12d4159a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:59008/0 (socket says 10.20.193.51:59008) 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12da3ce640 1 -- 10.20.193.51:0/1388822926 learned_addr learned my addr 10.20.193.51:0/1388822926 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d93cc640 1 --2- 10.20.193.51:0/1388822926 >> v2:10.20.193.125:3300/0 conn(0x7f12d40c9500 0x7f12d415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1388822926 >> v2:10.20.193.68:3300/0 conn(0x7f12d4153860 0x7f12d415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d93cc640 10 monclient(hunting): get_auth_request con 0x7f12d40c9500 auth_method 0 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d93cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d93cc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d93cc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.392 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d9bcd640 10 monclient(hunting): get_auth_request con 0x7f12d4153860 auth_method 0 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d9bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d9bcd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.379+0000 7f12d9bcd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 1 -- 10.20.193.51:0/1388822926 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101259638 0 0) 0x7f12d40c8180 con 0x7f12d40c7c50 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.393 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 10 monclient(hunting): my global_id is 5274 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.380+0000 7f12d8bcb640 1 -- 10.20.193.51:0/1388822926 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12bc003620 con 0x7f12d40c7c50 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_done global_id 5281 payload 274 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 1 -- 10.20.193.51:0/1388822926 >> v2:10.20.193.125:3300/0 conn(0x7f12d40c9500 msgr2=0x7f12d415d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1388822926 >> v2:10.20.193.125:3300/0 conn(0x7f12d40c9500 0x7f12d415d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 1 -- 10.20.193.51:0/1388822926 >> v1:10.20.193.51:6789/0 conn(0x7f12d40c7c50 legacy=0x7f12d4159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 1 monclient: found mon.noname-b 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.381+0000 7f12d9bcd640 1 -- 10.20.193.51:0/1388822926 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d40c8a20 con 0x7f12d4153860 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.382+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1388822926 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d4153860 0x7f12d415af10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12d415cf50 tx=0x7f12c402ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aef345270ec1719 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.394 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 1 -- 10.20.193.51:0/1388822926 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c403c070 con 0x7f12d4153860 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: dump: 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.395 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 1 -- 10.20.193.51:0/1388822926 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c402faf0 con 0x7f12d4153860 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 5 monclient: authenticate success, global_id 5281 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 1 -- 10.20.193.51:0/1388822926 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4037440 con 0x7f12d4153860 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 4 set_mon_vals no callback set 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.396 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12d8bcb640 10 monclient: dump: 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.397 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 10 monclient: shutdown 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 -- 10.20.193.51:0/1388822926 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d4153860 msgr2=0x7f12d415af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1388822926 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d4153860 0x7f12d415af10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12d415cf50 tx=0x7f12c402ede0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 -- 10.20.193.51:0/1388822926 shutdown_connections 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1388822926 >> v2:10.20.193.125:3300/0 conn(0x7f12d40c9500 0x7f12d415d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1388822926 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d4153860 0x7f12d415af10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 -- 10.20.193.51:0/1388822926 >> 10.20.193.51:0/1388822926 conn(0x7f12d40c9d50 msgr2=0x7f12d40ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 -- 10.20.193.51:0/1388822926 shutdown_connections 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 1 -- 10.20.193.51:0/1388822926 wait complete. 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.384+0000 7f12dbe58640 10 monclient: monmap: 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.398 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 librados: starting msgr at 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 librados: starting objecter 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 Processor -- start 2026-02-22T21:42:51.399 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 -- start start 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 librados: setting wanted keys 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 librados: calling monclient init 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient: init 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f12d40c7c50 0x7f12d41f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-c con 0x7f12d40c7c50 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 0x7f12d41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-a con 0x7f12d4153860 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d41f4e10 0x7f12d4159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): picked mon.noname-b con 0x7f12d41f4e10 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.400 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f12d415b450 con 0x7f12d4153860 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f12d40c8fe0 con 0x7f12d41f4e10 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f12d4153c40 con 0x7f12d40c7c50 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12dbe58640 10 monclient(hunting): authenticate will time out at 1446.965600s 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12d93cc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 0x7f12d41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.385+0000 7f12da3ce640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d41f4e10 0x7f12d4159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 0x7f12d41599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35582/0 (socket says 10.20.193.51:35582) 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 1 -- 10.20.193.51:0/1919000577 learned_addr learned my addr 10.20.193.51:0/1919000577 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): get_auth_request con 0x7f12d4153860 auth_method 0 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.401 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): get_auth_request con 0x7f12d41f4e10 auth_method 0 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f12d40c7c50 0x7f12d41f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d93cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.402 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12da3ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): get_auth_request con 0x7f12d40c7c50 auth_method 0 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.386+0000 7f12d9bcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.403 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 10 monclient(hunting): handle_auth_done global_id 5277 payload 824 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 -- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f12d40c7c50 msgr2=0x7f12d41f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f12d40c7c50 0x7f12d41f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 -- 10.20.193.51:0/1919000577 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d41f4e10 msgr2=0x7f12d4159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f12d41f4e10 0x7f12d4159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.387+0000 7f12d93cc640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d4158c20 con 0x7f12d4153860 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.389+0000 7f12d93cc640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 0x7f12d41599a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ebe0 tx=0x7f12c800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.389+0000 7f12c27fc640 10 client.?.objecter ms_handle_connect 0x7f12d4153860 2026-02-22T21:42:51.404 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.389+0000 7f12c27fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8017070 con 0x7f12d4153860 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: dump: 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.408 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 5 monclient: authenticate success, global_id 5277 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c800cc40 con 0x7f12d4153860 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 10 monclient: _renew_subs 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d414f7e0 con 0x7f12d4153860 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8013470 con 0x7f12d4153860 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c3fff640 4 set_mon_vals no callback set 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12c27fc640 10 monclient: dump: 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.410 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 10 client.5277.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 10 monclient: _renew_subs 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d41591a0 con 0x7f12d4153860 2026-02-22T21:42:51.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.391+0000 7f12dbe58640 1 librados: init done 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 10 client.5277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 15 client.5277.objecter _session_command_op_assign -1 1 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 20 client.5277.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 10 client.5277.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.392+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f12d40caad0 con 0x7f12d4153860 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.403+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f12c8024060 con 0x7f12d4153860 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.403+0000 7f12c27fc640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f12a4078e50 0x7f12a407b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f12a4078e50 0x7f12a407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12d9bcd640 10 monclient: get_auth_request con 0x7f12a4078e50 auth_method 0 2026-02-22T21:42:51.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 10 client.5277.objecter ms_handle_connect 0x7f12a4078e50 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12d9bcd640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f12a4078e50 0x7f12a407b310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f12c4039000 tx=0x7f12c4033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f12c805dd60 con 0x7f12d4153860 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 10 client.5277.objecter ms_dispatch2 0x7f12d4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 3 client.5277.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 3 client.5277.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 10 client.5277.objecter checking command 1 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f12a4081330 0x7f12a4083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter _get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 15 client.5277.objecter _session_command_op_remove -1 1 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter _get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 15 client.5277.objecter _session_command_op_assign 10 1 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 4 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 10 client.5277.objecter _send_command 1 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12c8016ec0 con 0x7f12a4081330 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.404+0000 7f12c27fc640 20 client.5277.objecter dump_active .. 0 homeless 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.405+0000 7f12da3ce640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f12a4081330 0x7f12a4083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.405+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f12c805e120 con 0x7f12d4153860 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.405+0000 7f12c27fc640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.405+0000 7f12da3ce640 10 monclient: get_auth_request con 0x7f12a4081330 auth_method 0 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.406+0000 7f12c27fc640 10 client.5277.objecter ms_handle_connect 0x7f12a4081330 2026-02-22T21:42:51.414 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.406+0000 7f12da3ce640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f12a4081330 0x7f12a4083750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.417+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== osd.10 v2:10.20.193.125:6800/1951583296 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f12c8016ec0 con 0x7f12a4081330 2026-02-22T21:42:51.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.417+0000 7f12c27fc640 10 client.5277.objecter ms_dispatch2 0x7f12d4000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:51.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.417+0000 7f12c27fc640 10 client.5277.objecter _finish_command 1 = system:0 2026-02-22T21:42:51.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.417+0000 7f12c27fc640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 4 2026-02-22T21:42:51.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.417+0000 7f12c27fc640 15 client.5277.objecter _session_command_op_remove 10 1 2026-02-22T21:42:51.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 10 client.5277.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:51.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 15 client.5277.objecter _session_command_op_assign -1 2 2026-02-22T21:42:51.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 20 client.5277.objecter _get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 20 client.5277.objecter _get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 15 client.5277.objecter _session_command_op_remove -1 2 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 20 client.5277.objecter get_session s=0x7f12a4080c60 osd=10 3 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 15 client.5277.objecter _session_command_op_assign 10 2 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 4 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 10 client.5277.objecter _send_command 2 2026-02-22T21:42:51.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.422+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 --> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f12d415af10 con 0x7f12a4081330 2026-02-22T21:42:51.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12c27fc640 1 -- 10.20.193.51:0/1919000577 <== osd.10 v2:10.20.193.125:6800/1951583296 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f12d415af10 con 0x7f12a4081330 2026-02-22T21:42:51.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12c27fc640 10 client.5277.objecter ms_dispatch2 0x7f12d4000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:51.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12c27fc640 10 client.5277.objecter _finish_command 2 = system:0 2026-02-22T21:42:51.438 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12c27fc640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 4 2026-02-22T21:42:51.439 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12c27fc640 15 client.5277.objecter _session_command_op_remove 10 2 2026-02-22T21:42:51.439 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:51.439 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:51.439 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 10 librados: watch_flush enter 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 10 librados: watch_flush exit 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 10 client.5277.objecter close_session for osd.10 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f12a4081330 msgr2=0x7f12a4083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.125:6800/1951583296,v1:10.20.193.125:6801/1951583296] conn(0x7f12a4081330 0x7f12a4083750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 20 client.5277.objecter put_session s=0x7f12a4080c60 osd=10 1 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 20 client.5277.objecter shutdown clearing up homeless session... 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 10 client.5277.objecter successfully canceled tick 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f12a4078e50 msgr2=0x7f12a407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f12a4078e50 0x7f12a407b310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f12c4039000 tx=0x7f12c4033000 comp rx=0 tx=0).stop 2026-02-22T21:42:51.441 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 10 monclient: shutdown 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 msgr2=0x7f12d41599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 --2- 10.20.193.51:0/1919000577 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f12d4153860 0x7f12d41599a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ebe0 tx=0x7f12c800c520 comp rx=0 tx=0).stop 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12da3ce640 1 -- 10.20.193.51:0/1919000577 reap_dead start 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 shutdown_connections 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 >> 10.20.193.51:0/1919000577 conn(0x7f12d40c9d50 msgr2=0x7f12d40ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 shutdown_connections 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 -- 10.20.193.51:0/1919000577 wait complete. 2026-02-22T21:42:51.442 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.437+0000 7f12dbe58640 1 librados: shutdown 2026-02-22T21:42:51.446 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.513+0000 7f39969fc640 10 monclient: get_monmap_and_config 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.513+0000 7f39969fc640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.513+0000 7f39969fc640 10 monclient: monmap: 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:51.514597+0000 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:51.514597+0000 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:51.515 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.516 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 1 Processor -- start 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 1 -- start start 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 10 monclient: init 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 10 monclient: _add_conns ranks=[1,2,0,5,4,3] 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f3990153a60 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.514+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f399006d4b0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3990074470 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3990159c40 con 0x7f3990074470 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3990159f20 con 0x7f3990153a60 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f399015a200 con 0x7f399006d4b0 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39969fc640 10 monclient(hunting): authenticate will time out at 1447.095080s 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39961fb640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f3990074470 0x7f3990074880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:59024/0 (socket says 10.20.193.51:59024) 2026-02-22T21:42:51.517 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.515+0000 7f39961fb640 1 -- 10.20.193.51:0/1261417420 learned_addr learned my addr 10.20.193.51:0/1261417420 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501994388 0 0) 0x7f3990159c40 con 0x7f3990074470 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): my global_id is 5280 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3978003620 con 0x7f3990074470 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476727139 0 0) 0x7f3990159f20 con 0x7f3990153a60 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): my global_id is 5287 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3990159c40 con 0x7f3990153a60 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570953418 0 0) 0x7f399015a200 con 0x7f399006d4b0 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.518 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 10 monclient(hunting): my global_id is 5270 2026-02-22T21:42:51.519 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.516+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3990159f20 con 0x7f399006d4b0 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.1 v1:10.20.193.68:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3289098781 0 0) 0x7f3990159c40 con 0x7f3990153a60 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 >> v1:10.20.193.125:6789/0 conn(0x7f399006d4b0 legacy=0x7f399006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 >> v1:10.20.193.51:6789/0 conn(0x7f3990074470 legacy=0x7f3990074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 monclient: found mon.noname-b-legacy 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _send_mon_message to mon.noname-b-legacy at v1:10.20.193.68:6789/0 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> v1:10.20.193.68:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399015a4e0 con 0x7f3990153a60 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.520 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.1 v1:10.20.193.68:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 426021516 0 0) 0x7f3980002ee0 con 0x7f3990153a60 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: got monmap 1 from mon.noname-b-legacy (according to old e1) 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: dump: 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 monclient: mon.b has (v2) addrs [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] but i'm connected to v1:10.20.193.68:6789/0, reconnecting 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 >> v1:10.20.193.68:6789/0 conn(0x7f3990153a60 legacy=0x7f39900c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3978003fe0 0x7f3978004900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): picked mon.c con 0x7f3978003fe0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3978004e40 0x7f39780052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.521 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): picked mon.b con 0x7f3978004e40 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f397800bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): picked mon.a con 0x7f3978009ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f397800c5c0 con 0x7f3978009ab0 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f397800c770 con 0x7f3978004e40 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f397800c970 con 0x7f3978003fe0 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.518+0000 7f39949f8640 10 monclient(hunting): _finish_auth 0 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3978003fe0 0x7f3978004900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.522 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f397800bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): get_auth_request con 0x7f3978003fe0 auth_method 0 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): get_auth_request con 0x7f3978009ab0 auth_method 0 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39951f9640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3978004e40 0x7f39780052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:51.524 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39951f9640 10 monclient(hunting): get_auth_request con 0x7f3978004e40 auth_method 0 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39951f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39951f9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.519+0000 7f39951f9640 10 monclient(hunting): _init_auth already have auth, reseting 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 10 monclient(hunting): handle_auth_done global_id 5287 payload 293 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 -- 10.20.193.51:0/1261417420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3978003fe0 msgr2=0x7f3978004900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f3978003fe0 0x7f3978004900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 -- 10.20.193.51:0/1261417420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3978004e40 msgr2=0x7f39780052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f3978004e40 0x7f39780052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 monclient: found mon.a 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 1 -- 10.20.193.51:0/1261417420 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3990159c40 con 0x7f3978009ab0 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.525 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39961fb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39969fc640 5 monclient: authenticate success, global_id 5287 2026-02-22T21:42:51.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.520+0000 7f39969fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:51.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.521+0000 7f39961fb640 1 -- 10.20.193.51:0/1261417420 reap_dead start 2026-02-22T21:42:51.526 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.521+0000 7f39961fb640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f397800bea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c4c0 tx=0x7f398c02f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794ea0ce4d05ebde server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c00c0e0 con 0x7f3978009ab0 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: dump: 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.528 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398c037630 con 0x7f3978009ab0 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 1 -- 10.20.193.51:0/1261417420 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c0379e0 con 0x7f3978009ab0 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 4 set_mon_vals no callback set 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39949f8640 10 monclient: dump: 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.529 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 10 monclient: shutdown 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 -- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 msgr2=0x7f397800bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f397800bea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c4c0 tx=0x7f398c02f6b0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 -- 10.20.193.51:0/1261417420 shutdown_connections 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 --2- 10.20.193.51:0/1261417420 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f397800bea0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 -- 10.20.193.51:0/1261417420 >> 10.20.193.51:0/1261417420 conn(0x7f39900c9c40 msgr2=0x7f39900ca050 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 -- 10.20.193.51:0/1261417420 shutdown_connections 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.523+0000 7f39969fc640 1 -- 10.20.193.51:0/1261417420 wait complete. 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 10 monclient: monmap: 2026-02-22T21:42:51.530 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 librados: starting msgr at 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 librados: starting objecter 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 Processor -- start 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 -- start start 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 librados: setting wanted keys 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 1 librados: calling monclient init 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.524+0000 7f39969fc640 10 monclient: init 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:51.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3978009ab0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39900cb7b0 0x7f3990154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-b con 0x7f39900cb7b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f39901548d0 0x7f3990156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): picked mon.noname-c con 0x7f39901548d0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f3990153e40 con 0x7f3978009ab0 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f399015a850 con 0x7f39900cb7b0 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f3990153fc0 con 0x7f39901548d0 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39969fc640 10 monclient(hunting): authenticate will time out at 1447.105120s 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35590/0 (socket says 10.20.193.51:35590) 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 1 -- 10.20.193.51:0/3232981513 learned_addr learned my addr 10.20.193.51:0/3232981513 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): get_auth_request con 0x7f3978009ab0 auth_method 0 2026-02-22T21:42:51.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39961fb640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f39901548d0 0x7f3990156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39951f9640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39900cb7b0 0x7f3990154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39959fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39951f9640 10 monclient(hunting): get_auth_request con 0x7f39900cb7b0 auth_method 0 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39951f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39951f9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39951f9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39961fb640 10 monclient(hunting): get_auth_request con 0x7f39901548d0 auth_method 0 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39961fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39961fb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.525+0000 7f39961fb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39961fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39951f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 10 monclient(hunting): handle_auth_done global_id 5283 payload 824 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 -- 10.20.193.51:0/3232981513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f39901548d0 msgr2=0x7f3990156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f39901548d0 0x7f3990156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 -- 10.20.193.51:0/3232981513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39900cb7b0 msgr2=0x7f3990154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39900cb7b0 0x7f3990154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.526+0000 7f39959fa640 1 -- 10.20.193.51:0/3232981513 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39901f7c10 con 0x7f3978009ab0 2026-02-22T21:42:51.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.527+0000 7f39959fa640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3980007fd0 tx=0x7f3980004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.527+0000 7f397e7fc640 10 client.?.objecter ms_handle_connect 0x7f3978009ab0 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.527+0000 7f397e7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980017020 con 0x7f3978009ab0 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: dump: 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398000f7b0 con 0x7f3978009ab0 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398000fba0 con 0x7f3978009ab0 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 5 monclient: authenticate success, global_id 5283 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397ffff640 4 set_mon_vals no callback set 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 10 monclient: _renew_subs 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39901f7650 con 0x7f3978009ab0 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f397e7fc640 10 monclient: dump: 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.537 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 10 client.5283.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 10 monclient: _renew_subs 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3990156140 con 0x7f3978009ab0 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.529+0000 7f39969fc640 1 librados: init done 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.530+0000 7f39969fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.530+0000 7f39969fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.530+0000 7f39969fc640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.530+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399006d940 con 0x7f3978009ab0 2026-02-22T21:42:51.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.541+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f398001c400 con 0x7f3978009ab0 2026-02-22T21:42:51.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.541+0000 7f397e7fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 0x7f396007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.541+0000 7f39951f9640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 0x7f396007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.541+0000 7f39951f9640 10 monclient: get_auth_request con 0x7f3960078e50 auth_method 0 2026-02-22T21:42:51.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f39951f9640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 0x7f396007b310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f39900cc200 tx=0x7f39840026e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 10 client.5283.objecter ms_handle_connect 0x7f3960078e50 2026-02-22T21:42:51.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f398005e580 con 0x7f3978009ab0 2026-02-22T21:42:51.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 10 client.5283.objecter ms_dispatch2 0x7f3990000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:51.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 3 client.5283.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:51.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 3 client.5283.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:51.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.542+0000 7f397e7fc640 20 client.5283.objecter dump_active .. 0 homeless 2026-02-22T21:42:51.647 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.646+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f3980021020 con 0x7f3978009ab0 2026-02-22T21:42:51.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.646+0000 7f397e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.648 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.646+0000 7f397e7fc640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:51.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.728+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "10", "target": ["mon-mgr", ""]}) -- 0x7f39900c7530 con 0x7f3960078e50 2026-02-22T21:42:51.730 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.728+0000 7f397e7fc640 1 -- 10.20.193.51:0/3232981513 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 10 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7f39900c7530 con 0x7f3960078e50 2026-02-22T21:42:51.730 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 10 to deep-scrub 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 10 librados: watch_flush enter 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 10 librados: watch_flush exit 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 20 client.5283.objecter shutdown clearing up homeless session... 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 10 client.5283.objecter successfully canceled tick 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 msgr2=0x7f396007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 0x7f396007b310 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f39900cc200 tx=0x7f39840026e0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 10 monclient: shutdown 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 msgr2=0x7f39901f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3980007fd0 tx=0x7f3980004b80 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 shutdown_connections 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f3960078e50 0x7f396007b310 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f39901548d0 0x7f3990156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f39900cb7b0 0x7f3990154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 --2- 10.20.193.51:0/3232981513 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f3978009ab0 0x7f39901f6d10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 >> 10.20.193.51:0/3232981513 conn(0x7f39900c9c40 msgr2=0x7f399014e450 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 shutdown_connections 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 -- 10.20.193.51:0/3232981513 wait complete. 2026-02-22T21:42:51.731 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.729+0000 7f39969fc640 1 librados: shutdown 2026-02-22T21:42:51.739 INFO:tasks.ceph:Scrubbing osd.14 2026-02-22T21:42:51.739 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:51.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 10 monclient: get_monmap_and_config 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 10 monclient: monmap: 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:51.808057+0000 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:51.808057+0000 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:51.809 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:51.810 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 1 Processor -- start 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 1 -- start start 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.807+0000 7fb214074640 10 monclient: init 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient: _add_conns ranks=[0,1,4,2,5,3] 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb20c153860 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fb20c0c9500 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7fb20c0c7c50 0x7fb20c1643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): picked mon.noname-b con 0x7fb20c0c7c50 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:51.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb20c0c8180 con 0x7fb20c153860 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb20c0c8460 con 0x7fb20c0c9500 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7fb20c0c8740 con 0x7fb20c0c7c50 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb214074640 10 monclient(hunting): authenticate will time out at 1447.388398s 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb211de9640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7fb20c153860 0x7fb20c15af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:59030/0 (socket says 10.20.193.51:59030) 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb211de9640 1 -- 10.20.193.51:0/556585137 learned_addr learned my addr 10.20.193.51:0/556585137 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.808+0000 7fb2125ea640 1 --2- 10.20.193.51:0/556585137 >> v2:10.20.193.68:3300/0 conn(0x7fb20c0c7c50 0x7fb20c1643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): get_auth_request con 0x7fb20c0c7c50 auth_method 0 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 1 -- 10.20.193.51:0/556585137 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068438433 0 0) 0x7fb20c0c8180 con 0x7fb20c153860 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 10 monclient(hunting): my global_id is 5286 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.809+0000 7fb210de7640 1 -- 10.20.193.51:0/556585137 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1f4003620 con 0x7fb20c153860 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 10 monclient(hunting): handle_auth_done global_id 5293 payload 274 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 1 -- 10.20.193.51:0/556585137 >> v1:10.20.193.68:6789/0 conn(0x7fb20c0c9500 legacy=0x7fb20c15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 1 -- 10.20.193.51:0/556585137 >> v1:10.20.193.51:6789/0 conn(0x7fb20c153860 legacy=0x7fb20c15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.813 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 1 monclient: found mon.noname-b 2026-02-22T21:42:51.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:51.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.810+0000 7fb2125ea640 1 -- 10.20.193.51:0/556585137 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c0c8a20 con 0x7fb20c0c7c50 2026-02-22T21:42:51.814 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.811+0000 7fb2125ea640 1 --2- 10.20.193.51:0/556585137 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1643a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20802f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb2aace350d1e5f9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.819 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 1 -- 10.20.193.51:0/556585137 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20803c070 con 0x7fb20c0c7c50 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: dump: 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.820 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 1 -- 10.20.193.51:0/556585137 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208037440 con 0x7fb20c0c7c50 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 1 -- 10.20.193.51:0/556585137 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208037810 con 0x7fb20c0c7c50 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb210de7640 10 monclient: dump: 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb214074640 5 monclient: authenticate success, global_id 5293 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb214074640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.813+0000 7fb214074640 4 set_mon_vals no callback set 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 10 monclient: shutdown 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- 10.20.193.51:0/556585137 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 msgr2=0x7fb20c1643a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 --2- 10.20.193.51:0/556585137 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1643a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20802f1e0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- 10.20.193.51:0/556585137 shutdown_connections 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 --2- 10.20.193.51:0/556585137 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1643a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- 10.20.193.51:0/556585137 >> 10.20.193.51:0/556585137 conn(0x7fb20c0c9d50 msgr2=0x7fb20c0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- 10.20.193.51:0/556585137 shutdown_connections 2026-02-22T21:42:51.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- 10.20.193.51:0/556585137 wait complete. 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 10 monclient: monmap: 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:51.824 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 librados: starting msgr at 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 librados: starting objecter 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 Processor -- start 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 -- start start 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 librados: setting wanted keys 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 1 librados: calling monclient init 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.814+0000 7fb214074640 10 monclient: init 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): picked mon.noname-b con 0x7fb20c0c7c50 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 0x7fb20c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): picked mon.noname-a con 0x7fb20c153860 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb20c15d690 0x7fb20c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): picked mon.noname-c con 0x7fb20c15d690 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c153c40 con 0x7fb20c153860 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c0c8fe0 con 0x7fb20c0c7c50 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c153dc0 con 0x7fb20c15d690 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb214074640 10 monclient(hunting): authenticate will time out at 1447.395248s 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 0x7fb20c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 0x7fb20c1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35606/0 (socket says 10.20.193.51:35606) 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 1 -- 10.20.193.51:0/3731146832 learned_addr learned my addr 10.20.193.51:0/3731146832 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): get_auth_request con 0x7fb20c153860 auth_method 0 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb211de9640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2125ea640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb20c15d690 0x7fb20c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb211de9640 10 monclient(hunting): get_auth_request con 0x7fb20c0c7c50 auth_method 0 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb211de9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb211de9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb211de9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.815+0000 7fb2115e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): get_auth_request con 0x7fb20c15d690 auth_method 0 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb211de9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb211de9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb211de9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.816+0000 7fb2125ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 10 monclient(hunting): handle_auth_done global_id 5289 payload 824 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 -- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb20c15d690 msgr2=0x7fb20c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fb20c15d690 0x7fb20c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 -- 10.20.193.51:0/3731146832 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 msgr2=0x7fb20c1445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fb20c0c7c50 0x7fb20c1445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.829 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.817+0000 7fb2115e8640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c145860 con 0x7fb20c153860 2026-02-22T21:42:51.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.818+0000 7fb2115e8640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 0x7fb20c1f23c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c530 tx=0x7fb20000ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.818+0000 7fb1fa7fc640 10 client.?.objecter ms_handle_connect 0x7fb20c153860 2026-02-22T21:42:51.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.818+0000 7fb1fa7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:51.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.819+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200016020 con 0x7fb20c153860 2026-02-22T21:42:51.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: dump: 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.833 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb200012850 con 0x7fb20c153860 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200012c40 con 0x7fb20c153860 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 5 monclient: authenticate success, global_id 5289 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 monclient: _renew_subs 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20c144ce0 con 0x7fb20c153860 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fbfff640 4 set_mon_vals no callback set 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb1fa7fc640 10 monclient: dump: 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.835 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 client.5289.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 monclient: _renew_subs 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20c1452a0 con 0x7fb20c153860 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 1 librados: init done 2026-02-22T21:42:51.836 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 client.5289.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 15 client.5289.objecter _session_command_op_assign -1 1 2026-02-22T21:42:51.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 20 client.5289.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:51.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 client.5289.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.837 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.820+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb20c0c5e80 con 0x7fb20c153860 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.831+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fb20001b400 con 0x7fb20c153860 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.832+0000 7fb1fa7fc640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1dc078e00 0x7fb1dc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.832+0000 7fb211de9640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1dc078e00 0x7fb1dc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.832+0000 7fb211de9640 10 monclient: get_auth_request con 0x7fb1dc078e00 auth_method 0 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.832+0000 7fb211de9640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1dc078e00 0x7fb1dc07b2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc007b60 tx=0x7fb1fc007ab0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.839 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.832+0000 7fb1fa7fc640 10 client.5289.objecter ms_handle_connect 0x7fb1dc078e00 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fb200012de0 con 0x7fb20c153860 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 10 client.5289.objecter ms_dispatch2 0x7fb20c000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 3 client.5289.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 3 client.5289.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 10 client.5289.objecter checking command 1 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7fb1dc0812e0 0x7fb1dc083700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter _get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 15 client.5289.objecter _session_command_op_remove -1 1 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter _get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 15 client.5289.objecter _session_command_op_assign 14 1 2026-02-22T21:42:51.840 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 4 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 10 client.5289.objecter _send_command 1 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb20000ae50 con 0x7fb1dc0812e0 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.833+0000 7fb1fa7fc640 20 client.5289.objecter dump_active .. 0 homeless 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.834+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb20005e6b0 con 0x7fb20c153860 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.834+0000 7fb1fa7fc640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.834+0000 7fb2125ea640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7fb1dc0812e0 0x7fb1dc083700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.834+0000 7fb2125ea640 10 monclient: get_auth_request con 0x7fb1dc0812e0 auth_method 0 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.836+0000 7fb1fa7fc640 10 client.5289.objecter ms_handle_connect 0x7fb1dc0812e0 2026-02-22T21:42:51.841 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.836+0000 7fb2125ea640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7fb1dc0812e0 0x7fb1dc083700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.14 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.845+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== osd.14 v2:10.20.193.125:6816/498199928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7fb20000ae50 con 0x7fb1dc0812e0 2026-02-22T21:42:51.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.845+0000 7fb1fa7fc640 10 client.5289.objecter ms_dispatch2 0x7fb20c000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:51.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.845+0000 7fb1fa7fc640 10 client.5289.objecter _finish_command 1 = system:0 2026-02-22T21:42:51.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.845+0000 7fb1fa7fc640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 4 2026-02-22T21:42:51.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.845+0000 7fb1fa7fc640 15 client.5289.objecter _session_command_op_remove 14 1 2026-02-22T21:42:51.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 10 client.5289.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:51.853 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 15 client.5289.objecter _session_command_op_assign -1 2 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 20 client.5289.objecter _get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 20 client.5289.objecter _get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 15 client.5289.objecter _session_command_op_remove -1 2 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 20 client.5289.objecter get_session s=0x7fb1dc080c10 osd=14 3 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 15 client.5289.objecter _session_command_op_assign 14 2 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 4 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 10 client.5289.objecter _send_command 2 2026-02-22T21:42:51.854 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.852+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 --> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb20c15af10 con 0x7fb1dc0812e0 2026-02-22T21:42:51.867 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb1fa7fc640 1 -- 10.20.193.51:0/3731146832 <== osd.14 v2:10.20.193.125:6816/498199928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7fb20c15af10 con 0x7fb1dc0812e0 2026-02-22T21:42:51.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb1fa7fc640 10 client.5289.objecter ms_dispatch2 0x7fb20c000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:51.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb1fa7fc640 10 client.5289.objecter _finish_command 2 = system:0 2026-02-22T21:42:51.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb1fa7fc640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 4 2026-02-22T21:42:51.868 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb1fa7fc640 15 client.5289.objecter _session_command_op_remove 14 2 2026-02-22T21:42:51.868 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:51.869 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:51.869 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:51.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 10 librados: watch_flush enter 2026-02-22T21:42:51.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 10 librados: watch_flush exit 2026-02-22T21:42:51.870 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 10 client.5289.objecter close_session for osd.14 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7fb1dc0812e0 msgr2=0x7fb1dc083700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.125:6816/498199928,v1:10.20.193.125:6817/498199928] conn(0x7fb1dc0812e0 0x7fb1dc083700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 20 client.5289.objecter put_session s=0x7fb1dc080c10 osd=14 1 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 20 client.5289.objecter shutdown clearing up homeless session... 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 10 client.5289.objecter successfully canceled tick 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1dc078e00 msgr2=0x7fb1dc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fb1dc078e00 0x7fb1dc07b2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc007b60 tx=0x7fb1fc007ab0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 10 monclient: shutdown 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 msgr2=0x7fb20c1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.871 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 --2- 10.20.193.51:0/3731146832 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fb20c153860 0x7fb20c1f23c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c530 tx=0x7fb20000ca00 comp rx=0 tx=0).stop 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb2125ea640 1 -- 10.20.193.51:0/3731146832 reap_dead start 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 shutdown_connections 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 >> 10.20.193.51:0/3731146832 conn(0x7fb20c0c9d50 msgr2=0x7fb20c0ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 shutdown_connections 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 -- 10.20.193.51:0/3731146832 wait complete. 2026-02-22T21:42:51.872 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.866+0000 7fb214074640 1 librados: shutdown 2026-02-22T21:42:51.876 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2026-02-22T21:42:51.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.942+0000 7fbc28c05640 10 monclient: get_monmap_and_config 2026-02-22T21:42:51.944 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.942+0000 7fbc28c05640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.942+0000 7fbc28c05640 10 monclient: monmap: 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:51.943399+0000 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:51.943399+0000 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:51.945 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 Processor -- start 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 -- start start 2026-02-22T21:42:51.946 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient: init 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient: _add_conns ranks=[3,2,1,0,4,5] 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbc24150220 0x7fbc240c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-a con 0x7fbc24150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc24150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:51.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fbc2415c710 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2415a920 con 0x7fbc2415c710 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2415af60 con 0x7fbc24150600 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fbc2415b240 con 0x7fbc24150220 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.943+0000 7fbc28c05640 10 monclient(hunting): authenticate will time out at 1447.523742s 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fbc24150220 0x7fbc240c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc21d74640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7fbc24150600 0x7fbc2415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47644/0 (socket says 10.20.193.51:47644) 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc21d74640 1 -- 10.20.193.51:0/106335819 learned_addr learned my addr 10.20.193.51:0/106335819 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:51.950 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): get_auth_request con 0x7fbc24150220 auth_method 0 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.944+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 1 -- 10.20.193.51:0/106335819 <== mon.2 v1:10.20.193.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165969966 0 0) 0x7fbc2415af60 con 0x7fbc24150600 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 10 monclient(hunting): _negotiate 2026-02-22T21:42:51.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 10 monclient(hunting): my global_id is 5279 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc21573640 1 -- 10.20.193.51:0/106335819 --> v1:10.20.193.125:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc10003620 con 0x7fbc24150600 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 10 monclient(hunting): handle_auth_done global_id 5292 payload 274 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 1 -- 10.20.193.51:0/106335819 >> v1:10.20.193.125:6789/0 conn(0x7fbc24150600 legacy=0x7fbc2415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 1 -- 10.20.193.51:0/106335819 >> v1:10.20.193.68:6789/0 conn(0x7fbc2415c710 legacy=0x7fbc2406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.952 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.945+0000 7fbc22575640 1 -- 10.20.193.51:0/106335819 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2415b520 con 0x7fbc24150220 2026-02-22T21:42:51.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.946+0000 7fbc22575640 1 --2- 10.20.193.51:0/106335819 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc240c73a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbc180098a0 tx=0x7fbc1802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8a3a1f1bb35af2 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 1 -- 10.20.193.51:0/106335819 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1803c070 con 0x7fbc24150220 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: dump: 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.960 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.961 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 1 -- 10.20.193.51:0/106335819 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1802fc10 con 0x7fbc24150220 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc28c05640 5 monclient: authenticate success, global_id 5292 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 1 -- 10.20.193.51:0/106335819 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18037440 con 0x7fbc24150220 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc21573640 10 monclient: dump: 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.962 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc28c05640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.948+0000 7fbc28c05640 4 set_mon_vals no callback set 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 10 monclient: shutdown 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:51.963 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- 10.20.193.51:0/106335819 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 msgr2=0x7fbc240c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 --2- 10.20.193.51:0/106335819 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc240c73a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbc180098a0 tx=0x7fbc1802ef00 comp rx=0 tx=0).stop 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- 10.20.193.51:0/106335819 shutdown_connections 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 --2- 10.20.193.51:0/106335819 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc240c73a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- 10.20.193.51:0/106335819 >> 10.20.193.51:0/106335819 conn(0x7fbc240cd7b0 msgr2=0x7fbc240cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- 10.20.193.51:0/106335819 shutdown_connections 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- 10.20.193.51:0/106335819 wait complete. 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 10 monclient: build_initial_monmap 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 10 monclient: monmap: 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:51.964 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 librados: starting msgr at 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 librados: starting objecter 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 Processor -- start 2026-02-22T21:42:51.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 -- start start 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 librados: setting wanted keys 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 1 librados: calling monclient init 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.949+0000 7fbc28c05640 10 monclient: init 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-a con 0x7fbc24150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc24150600 0x7fbc240c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-b con 0x7fbc24150600 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:51.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc2415c710 0x7fbc2414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2415c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fbc240703b0 con 0x7fbc24150220 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2415bae0 con 0x7fbc24150600 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24070530 con 0x7fbc2415c710 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc28c05640 10 monclient(hunting): authenticate will time out at 1447.530378s 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35624/0 (socket says 10.20.193.51:35624) 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 1 -- 10.20.193.51:0/2143921243 learned_addr learned my addr 10.20.193.51:0/2143921243 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): get_auth_request con 0x7fbc24150220 auth_method 0 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc21d74640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc24150600 0x7fbc240c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22d76640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc2415c710 0x7fbc2414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.950+0000 7fbc22575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): get_auth_request con 0x7fbc24150600 auth_method 0 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): get_auth_request con 0x7fbc2415c710 auth_method 0 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc22d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.951+0000 7fbc21d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 10 monclient(hunting): handle_auth_done global_id 5295 payload 824 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 -- 10.20.193.51:0/2143921243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc2415c710 msgr2=0x7fbc2414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc2415c710 0x7fbc2414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 -- 10.20.193.51:0/2143921243 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc24150600 msgr2=0x7fbc240c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc24150600 0x7fbc240c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 10 monclient: _finish_hunting 0 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 monclient: found mon.noname-a 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:51.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.952+0000 7fbc22575640 1 -- 10.20.193.51:0/2143921243 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc240c1b10 con 0x7fbc24150220 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.953+0000 7fbc22575640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc1802f430 tx=0x7fbc180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.953+0000 7fbc0affd640 10 client.?.objecter ms_handle_connect 0x7fbc24150220 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.953+0000 7fbc0affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1803c040 con 0x7fbc24150220 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: dump: 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.971 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: _finish_auth 0 2026-02-22T21:42:51.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc18042630 con 0x7fbc24150220 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc28c05640 5 monclient: authenticate success, global_id 5295 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc28c05640 10 monclient: _renew_subs 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc28c05640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc240c1df0 con 0x7fbc24150220 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18042a20 con 0x7fbc24150220 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:51.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc20d72640 4 set_mon_vals no callback set 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc0affd640 10 monclient: dump: 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:51.974 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.954+0000 7fbc28c05640 10 client.5295.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 10 monclient: _renew_subs 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc240c23b0 con 0x7fbc24150220 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 1 librados: init done 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:51.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:51.955+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc240c73a0 con 0x7fbc24150220 2026-02-22T21:42:52.061 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.060+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fbc180413b0 con 0x7fbc24150220 2026-02-22T21:42:52.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.060+0000 7fbc0affd640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 0x7fbbf007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.061+0000 7fbc21d74640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 0x7fbbf007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.061+0000 7fbc21d74640 10 monclient: get_auth_request con 0x7fbbf0078e50 auth_method 0 2026-02-22T21:42:52.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.061+0000 7fbc0affd640 10 client.5295.objecter ms_handle_connect 0x7fbbf0078e50 2026-02-22T21:42:52.063 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.061+0000 7fbc21d74640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 0x7fbbf007b310 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc241f4fd0 tx=0x7fbc0c007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.071 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fbc1802fb50 con 0x7fbc24150220 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 10 client.5295.objecter ms_dispatch2 0x7fbc24000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 3 client.5295.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 3 client.5295.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 20 client.5295.objecter dump_active .. 0 homeless 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fbc18087050 con 0x7fbc24150220 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.072 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.069+0000 7fbc0affd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:52.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.152+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "14", "target": ["mon-mgr", ""]}) -- 0x7fbc24074da0 con 0x7fbbf0078e50 2026-02-22T21:42:52.154 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.152+0000 7fbc0affd640 1 -- 10.20.193.51:0/2143921243 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 14 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7fbc24074da0 con 0x7fbbf0078e50 2026-02-22T21:42:52.154 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 14 to deep-scrub 2026-02-22T21:42:52.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 10 librados: watch_flush enter 2026-02-22T21:42:52.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 10 librados: watch_flush exit 2026-02-22T21:42:52.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 20 client.5295.objecter shutdown clearing up homeless session... 2026-02-22T21:42:52.156 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 10 client.5295.objecter successfully canceled tick 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 msgr2=0x7fbbf007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 0x7fbbf007b310 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc241f4fd0 tx=0x7fbc0c007dd0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 10 monclient: shutdown 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 msgr2=0x7fbc2406eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbc1802f430 tx=0x7fbc180047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 shutdown_connections 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fbbf0078e50 0x7fbbf007b310 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fbc2415c710 0x7fbc2414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fbc24150600 0x7fbc240c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 --2- 10.20.193.51:0/2143921243 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fbc24150220 0x7fbc2406eea0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 >> 10.20.193.51:0/2143921243 conn(0x7fbc240cd7b0 msgr2=0x7fbc241591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 shutdown_connections 2026-02-22T21:42:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 -- 10.20.193.51:0/2143921243 wait complete. 2026-02-22T21:42:52.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.153+0000 7fbc28c05640 1 librados: shutdown 2026-02-22T21:42:52.162 INFO:tasks.ceph:Scrubbing osd.3 2026-02-22T21:42:52.163 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:52.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.238+0000 7f5035c05640 10 monclient: get_monmap_and_config 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.238+0000 7f5035c05640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.238+0000 7f5035c05640 10 monclient: monmap: 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:52.239215+0000 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:52.239215+0000 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:52.240 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 Processor -- start 2026-02-22T21:42:52.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 -- start start 2026-02-22T21:42:52.241 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient: init 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient: _add_conns ranks=[1,5,3,2,0,4] 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f5030153860 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f50300c9500 0x7f503015d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-c con 0x7f50300c9500 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f50300c7c50 0x7f503015d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-a con 0x7f50300c7c50 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50300c8180 con 0x7f5030153860 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f503015da90 con 0x7f50300c7c50 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f50300c8460 con 0x7f50300c9500 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.242 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.239+0000 7f5035c05640 10 monclient(hunting): authenticate will time out at 1447.819672s 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f50300c9500 0x7f503015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f50300c9500 0x7f503015d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36948/0 (socket says 10.20.193.51:36948) 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 1 -- 10.20.193.51:0/922352605 learned_addr learned my addr 10.20.193.51:0/922352605 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 10 monclient(hunting): get_auth_request con 0x7f50300c9500 auth_method 0 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.241+0000 7f502effd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.242+0000 7f502effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.242+0000 7f502effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.242+0000 7f502effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 10 monclient(hunting): handle_auth_done global_id 5285 payload 274 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 1 -- 10.20.193.51:0/922352605 >> v2:10.20.193.51:3300/0 conn(0x7f50300c7c50 msgr2=0x7f503015d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 1 --2- 10.20.193.51:0/922352605 >> v2:10.20.193.51:3300/0 conn(0x7f50300c7c50 0x7f503015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 1 -- 10.20.193.51:0/922352605 >> v1:10.20.193.68:6789/0 conn(0x7f5030153860 legacy=0x7f503015af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 1 monclient: found mon.noname-c 2026-02-22T21:42:52.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:52.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.243+0000 7f502effd640 1 -- 10.20.193.51:0/922352605 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50300c8740 con 0x7f50300c9500 2026-02-22T21:42:52.274 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.273+0000 7f502effd640 1 --2- 10.20.193.51:0/922352605 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c9500 0x7f503015d010 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f503015cf50 tx=0x7f502002eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7ea625dfa5502b server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 1 -- 10.20.193.51:0/922352605 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502003c070 con 0x7f50300c9500 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: dump: 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 1 -- 10.20.193.51:0/922352605 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f502002fb40 con 0x7f50300c9500 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 5 monclient: authenticate success, global_id 5285 2026-02-22T21:42:52.278 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 1 -- 10.20.193.51:0/922352605 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020037440 con 0x7f50300c9500 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 4 set_mon_vals no callback set 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f502e7fc640 10 monclient: dump: 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.279 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 10 monclient: shutdown 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 -- 10.20.193.51:0/922352605 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c9500 msgr2=0x7f503015d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 --2- 10.20.193.51:0/922352605 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c9500 0x7f503015d010 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f503015cf50 tx=0x7f502002eed0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 -- 10.20.193.51:0/922352605 shutdown_connections 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 --2- 10.20.193.51:0/922352605 >> v2:10.20.193.51:3300/0 conn(0x7f50300c7c50 0x7f503015d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 --2- 10.20.193.51:0/922352605 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c9500 0x7f503015d010 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 -- 10.20.193.51:0/922352605 >> 10.20.193.51:0/922352605 conn(0x7f50300c9d50 msgr2=0x7f50300ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 -- 10.20.193.51:0/922352605 shutdown_connections 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.277+0000 7f5035c05640 1 -- 10.20.193.51:0/922352605 wait complete. 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient: monmap: 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 librados: starting msgr at 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 librados: starting objecter 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 Processor -- start 2026-02-22T21:42:52.280 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 -- start start 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 librados: setting wanted keys 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 librados: calling monclient init 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient: init 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c7c50 0x7f50300c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-c con 0x7f50300c7c50 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5030153860 0x7f50300cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-a con 0x7f5030153860 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50300c5440 0x7f50300cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): picked mon.noname-b con 0x7f50300c5440 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f50300c8fe0 con 0x7f5030153860 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f50300c8d00 con 0x7f50300c5440 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5030153c40 con 0x7f50300c7c50 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.278+0000 7f5035c05640 10 monclient(hunting): authenticate will time out at 1447.858786s 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c7c50 0x7f50300c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50300c5440 0x7f50300cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c7c50 0x7f50300c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:36958/0 (socket says 10.20.193.51:36958) 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 1 -- 10.20.193.51:0/2172474455 learned_addr learned my addr 10.20.193.51:0/2172474455 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.281 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): get_auth_request con 0x7f50300c7c50 auth_method 0 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): get_auth_request con 0x7f50300c5440 auth_method 0 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.279+0000 7f502ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 10 monclient(hunting): handle_auth_done global_id 5305 payload 824 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 -- 10.20.193.51:0/2172474455 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c7c50 msgr2=0x7f50300c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f50300c7c50 0x7f50300c4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 -- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5030153860 msgr2=0x7f50300cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5030153860 0x7f50300cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 monclient: found mon.noname-b 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.282 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.281+0000 7f502ffff640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50301f39e0 con 0x7f50300c5440 2026-02-22T21:42:52.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.282+0000 7f502ffff640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50300c5440 0x7f50300cb850 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f502400be50 tx=0x7f502400bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.282+0000 7f500bfff640 10 client.?.objecter ms_handle_connect 0x7f50300c5440 2026-02-22T21:42:52.283 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.282+0000 7f500bfff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502400ca10 con 0x7f50300c5440 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: dump: 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5024010070 con 0x7f50300c5440 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502400ce10 con 0x7f50300c5440 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f502dffb640 4 set_mon_vals no callback set 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f500bfff640 10 monclient: dump: 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.285 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 5 monclient: authenticate success, global_id 5305 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 10 monclient: _renew_subs 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50301f3cc0 con 0x7f50300c5440 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 10 client.5305.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 10 monclient: _renew_subs 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50301f4280 con 0x7f50300c5440 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.283+0000 7f5035c05640 1 librados: init done 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 10 client.5305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 15 client.5305.objecter _session_command_op_assign -1 1 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 20 client.5305.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 10 client.5305.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.286 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.284+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f503015af10 con 0x7f50300c5440 2026-02-22T21:42:52.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.295+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f50240040f0 con 0x7f50300c5440 2026-02-22T21:42:52.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.296+0000 7f500bfff640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5000078e50 0x7f500007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.296+0000 7f502f7fe640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5000078e50 0x7f500007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.297 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.296+0000 7f502f7fe640 10 monclient: get_auth_request con 0x7f5000078e50 auth_method 0 2026-02-22T21:42:52.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.296+0000 7f500bfff640 10 client.5305.objecter ms_handle_connect 0x7f5000078e50 2026-02-22T21:42:52.298 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.296+0000 7f502f7fe640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5000078e50 0x7f500007b310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f501c004660 tx=0x7f501c009210 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f502405de70 con 0x7f50300c5440 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 10 client.5305.objecter ms_dispatch2 0x7f5030000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 3 client.5305.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 3 client.5305.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 10 client.5305.objecter checking command 1 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7f5000081330 0x7f5000083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter _get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 15 client.5305.objecter _session_command_op_remove -1 1 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter _get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 15 client.5305.objecter _session_command_op_assign 3 1 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 4 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 10 client.5305.objecter _send_command 1 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5000083e20 con 0x7f5000081330 2026-02-22T21:42:52.299 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.297+0000 7f500bfff640 20 client.5305.objecter dump_active .. 0 homeless 2026-02-22T21:42:52.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.298+0000 7f502effd640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7f5000081330 0x7f5000083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.300 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.298+0000 7f502effd640 10 monclient: get_auth_request con 0x7f5000081330 auth_method 0 2026-02-22T21:42:52.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.300+0000 7f500bfff640 10 client.5305.objecter ms_handle_connect 0x7f5000081330 2026-02-22T21:42:52.301 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.300+0000 7f502effd640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7f5000081330 0x7f5000083750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.310+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== osd.3 v2:10.20.193.140:6824/3567184175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f5000083e20 con 0x7f5000081330 2026-02-22T21:42:52.311 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.310+0000 7f500bfff640 10 client.5305.objecter ms_dispatch2 0x7f5030000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:52.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.310+0000 7f500bfff640 10 client.5305.objecter _finish_command 1 = system:0 2026-02-22T21:42:52.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.310+0000 7f500bfff640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 4 2026-02-22T21:42:52.312 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.310+0000 7f500bfff640 15 client.5305.objecter _session_command_op_remove 3 1 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 10 client.5305.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 15 client.5305.objecter _session_command_op_assign -1 2 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 20 client.5305.objecter _get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 20 client.5305.objecter _get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 15 client.5305.objecter _session_command_op_remove -1 2 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 20 client.5305.objecter get_session s=0x7f5000080c60 osd=3 3 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 15 client.5305.objecter _session_command_op_assign 3 2 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 4 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 10 client.5305.objecter _send_command 2 2026-02-22T21:42:52.318 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.316+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 --> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f50300c4770 con 0x7f5000081330 2026-02-22T21:42:52.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== osd.3 v2:10.20.193.140:6824/3567184175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+712 (crc 0 0 0) 0x7f50300c4770 con 0x7f5000081330 2026-02-22T21:42:52.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 10 client.5305.objecter ms_dispatch2 0x7f5030000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:52.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 10 client.5305.objecter _finish_command 2 = system:0 2026-02-22T21:42:52.331 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 4 2026-02-22T21:42:52.332 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 15 client.5305.objecter _session_command_op_remove 3 2 2026-02-22T21:42:52.332 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:52.332 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T21:42:52.332 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:52.333 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 10 librados: watch_flush enter 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 10 librados: watch_flush exit 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 10 client.5305.objecter close_session for osd.3 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7f5000081330 msgr2=0x7f5000083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.140:6824/3567184175,v1:10.20.193.140:6825/3567184175] conn(0x7f5000081330 0x7f5000083750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 20 client.5305.objecter put_session s=0x7f5000080c60 osd=3 1 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 20 client.5305.objecter shutdown clearing up homeless session... 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 10 client.5305.objecter successfully canceled tick 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 1 -- 10.20.193.51:0/2172474455 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f502405e230 con 0x7f50300c5440 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f500bfff640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:52.334 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5000078e50 msgr2=0x7f500007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5000078e50 0x7f500007b310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f501c004660 tx=0x7f501c009210 comp rx=0 tx=0).stop 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 10 monclient: shutdown 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50300c5440 msgr2=0x7f50300cb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 --2- 10.20.193.51:0/2172474455 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f50300c5440 0x7f50300cb850 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f502400be50 tx=0x7f502400bf50 comp rx=0 tx=0).stop 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f502ffff640 1 -- 10.20.193.51:0/2172474455 reap_dead start 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 shutdown_connections 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 >> 10.20.193.51:0/2172474455 conn(0x7f50300c9d50 msgr2=0x7f503015b680 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 shutdown_connections 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 -- 10.20.193.51:0/2172474455 wait complete. 2026-02-22T21:42:52.335 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.330+0000 7f5035c05640 1 librados: shutdown 2026-02-22T21:42:52.340 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 10 monclient: get_monmap_and_config 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 10 monclient: monmap: 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:52.406800+0000 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:52.406800+0000 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:52.407 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 1 Processor -- start 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 1 -- start start 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.406+0000 7f8e027f7640 10 monclient: init 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient: _add_conns ranks=[2,5,4,3,0,1] 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8dfc150220 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:52.408 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f8dfc15c710 0x7f8dfc0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-c con 0x7f8dfc15c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8dfc150600 0x7f8dfc06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-b con 0x7f8dfc150600 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dfc15a920 con 0x7f8dfc150220 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f8dfc15af60 con 0x7f8dfc150600 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f8dfc15b240 con 0x7f8dfc15c710 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e027f7640 10 monclient(hunting): authenticate will time out at 1447.987090s 2026-02-22T21:42:52.409 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f8dfc150600 0x7f8dfc06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8dfbfff640 1 --1- >> v1:10.20.193.125:6789/0 conn(0x7f8dfc150220 0x7f8dfc15c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.125:6789/0 says I am v1:10.20.193.51:47650/0 (socket says 10.20.193.51:47650) 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8dfbfff640 1 -- 10.20.193.51:0/2681436386 learned_addr learned my addr 10.20.193.51:0/2681436386 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): get_auth_request con 0x7f8dfc150600 auth_method 0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8dfb7fe640 1 --2- 10.20.193.51:0/2681436386 >> v2:10.20.193.125:3300/0 conn(0x7f8dfc15c710 0x7f8dfc0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.407+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dfc15c710 auth_method 0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_done global_id 5308 payload 274 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 1 -- 10.20.193.51:0/2681436386 >> v2:10.20.193.125:3300/0 conn(0x7f8dfc15c710 msgr2=0x7f8dfc0c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 1 --2- 10.20.193.51:0/2681436386 >> v2:10.20.193.125:3300/0 conn(0x7f8dfc15c710 0x7f8dfc0c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 1 -- 10.20.193.51:0/2681436386 >> v1:10.20.193.125:6789/0 conn(0x7f8dfc150220 legacy=0x7f8dfc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 1 monclient: found mon.noname-b 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.410 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.408+0000 7f8e00d6d640 1 -- 10.20.193.51:0/2681436386 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc15b520 con 0x7f8dfc150600 2026-02-22T21:42:52.411 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.409+0000 7f8e00d6d640 1 --2- 10.20.193.51:0/2681436386 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150600 0x7f8dfc06fca0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009d90 tx=0x7f8df002f270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f025a7f12fd6a721 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 1 -- 10.20.193.51:0/2681436386 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df003c070 con 0x7f8dfc150600 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: dump: 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.412 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 5 monclient: authenticate success, global_id 5308 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 1 -- 10.20.193.51:0/2681436386 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df0037440 con 0x7f8dfc150600 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 1 -- 10.20.193.51:0/2681436386 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0037810 con 0x7f8dfc150600 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 4 set_mon_vals no callback set 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8dfaffd640 10 monclient: dump: 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.413 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 10 monclient: shutdown 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 -- 10.20.193.51:0/2681436386 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150600 msgr2=0x7f8dfc06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2681436386 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150600 0x7f8dfc06fca0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8df0009d90 tx=0x7f8df002f270 comp rx=0 tx=0).stop 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 -- 10.20.193.51:0/2681436386 shutdown_connections 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2681436386 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150600 0x7f8dfc06fca0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2681436386 >> v2:10.20.193.125:3300/0 conn(0x7f8dfc15c710 0x7f8dfc0c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 -- 10.20.193.51:0/2681436386 >> 10.20.193.51:0/2681436386 conn(0x7f8dfc0cd7b0 msgr2=0x7f8dfc0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 -- 10.20.193.51:0/2681436386 shutdown_connections 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.411+0000 7f8e027f7640 1 -- 10.20.193.51:0/2681436386 wait complete. 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient: monmap: 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.415 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 librados: starting msgr at 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 librados: starting objecter 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 Processor -- start 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 -- start start 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 librados: setting wanted keys 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 librados: calling monclient init 2026-02-22T21:42:52.416 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient: init 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 0x7f8dfc14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-b con 0x7f8dfc150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 0x7f8dfc1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-a con 0x7f8dfc150600 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dfc15c710 0x7f8dfc06e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): picked mon.noname-c con 0x7f8dfc15c710 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc0701e0 con 0x7f8dfc150600 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.417 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc15bae0 con 0x7f8dfc150220 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc070360 con 0x7f8dfc15c710 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.412+0000 7f8e027f7640 10 monclient(hunting): authenticate will time out at 1447.992963s 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 0x7f8dfc1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 0x7f8dfc14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dfc15c710 0x7f8dfc06e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 0x7f8dfc1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35648/0 (socket says 10.20.193.51:35648) 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 1 -- 10.20.193.51:0/2361755768 learned_addr learned my addr 10.20.193.51:0/2361755768 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dfc150600 auth_method 0 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.418 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request con 0x7f8dfc150220 auth_method 0 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): get_auth_request con 0x7f8dfc15c710 auth_method 0 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.419 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.420 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.413+0000 7f8e00d6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.422 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_done global_id 5311 payload 824 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 -- 10.20.193.51:0/2361755768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dfc15c710 msgr2=0x7f8dfc06e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dfc15c710 0x7f8dfc06e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 -- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 msgr2=0x7f8dfc1f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 0x7f8dfc1f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 monclient: found mon.noname-b 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.414+0000 7f8dfbfff640 1 -- 10.20.193.51:0/2361755768 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc06d000 con 0x7f8dfc150220 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.415+0000 7f8dfbfff640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 0x7f8dfc14d4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c500 tx=0x7f8de800c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.423 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.415+0000 7f8df8ff9640 10 client.?.objecter ms_handle_connect 0x7f8dfc150220 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.415+0000 7f8df8ff9640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8017070 con 0x7f8dfc150220 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: dump: 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.424 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 5 monclient: authenticate success, global_id 5311 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _renew_subs 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dfc06d2e0 con 0x7f8dfc150220 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 client.5311.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.425 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _renew_subs 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dfc06d8a0 con 0x7f8dfc150220 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 1 librados: init done 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8005de0 con 0x7f8dfc150220 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8021060 con 0x7f8dfc150220 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8df8ff9640 10 monclient: dump: 2026-02-22T21:42:52.426 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8dfa7fc640 4 set_mon_vals no callback set 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.427 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.428 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.417+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dfc074690 con 0x7f8dfc150220 2026-02-22T21:42:52.527 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.525+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f8de8024080 con 0x7f8dfc150220 2026-02-22T21:42:52.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.526+0000 7f8df8ff9640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 0x7f8dcc07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.528 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.526+0000 7f8dfb7fe640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 0x7f8dcc07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.526+0000 7f8dfb7fe640 10 monclient: get_auth_request con 0x7f8dcc078e00 auth_method 0 2026-02-22T21:42:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.526+0000 7f8df8ff9640 10 client.5311.objecter ms_handle_connect 0x7f8dcc078e00 2026-02-22T21:42:52.529 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.526+0000 7f8dfb7fe640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 0x7f8dcc07b2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc1f8440 tx=0x7f8dec009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.527+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f8de805d4d0 con 0x7f8dfc150220 2026-02-22T21:42:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.527+0000 7f8df8ff9640 10 client.5311.objecter ms_dispatch2 0x7f8dfc000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.527+0000 7f8df8ff9640 3 client.5311.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.527+0000 7f8df8ff9640 3 client.5311.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:52.530 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.527+0000 7f8df8ff9640 20 client.5311.objecter dump_active .. 0 homeless 2026-02-22T21:42:52.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.533+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f8dfc074690 con 0x7f8dfc150220 2026-02-22T21:42:52.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.533+0000 7f8df8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.533+0000 7f8df8ff9640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:52.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.616+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f8dfc06fca0 con 0x7f8dcc078e00 2026-02-22T21:42:52.617 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.616+0000 7f8df8ff9640 1 -- 10.20.193.51:0/2361755768 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8dfc06fca0 con 0x7f8dcc078e00 2026-02-22T21:42:52.617 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 3 to deep-scrub 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 10 librados: watch_flush enter 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 10 librados: watch_flush exit 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 20 client.5311.objecter shutdown clearing up homeless session... 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 10 client.5311.objecter successfully canceled tick 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 msgr2=0x7f8dcc07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 0x7f8dcc07b2c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc1f8440 tx=0x7f8dec009290 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 10 monclient: shutdown 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 msgr2=0x7f8dfc14d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 0x7f8dfc14d4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c500 tx=0x7f8de800c9d0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 shutdown_connections 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f8dcc078e00 0x7f8dcc07b2c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f8dfc15c710 0x7f8dfc06e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f8dfc150600 0x7f8dfc1f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 --2- 10.20.193.51:0/2361755768 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f8dfc150220 0x7f8dfc14d4a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 >> 10.20.193.51:0/2361755768 conn(0x7f8dfc0cd7b0 msgr2=0x7f8dfc0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 shutdown_connections 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 -- 10.20.193.51:0/2361755768 wait complete. 2026-02-22T21:42:52.618 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.617+0000 7f8e027f7640 1 librados: shutdown 2026-02-22T21:42:52.626 INFO:tasks.ceph:Scrubbing osd.7 2026-02-22T21:42:52.626 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:52.695 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.694+0000 7f5f88a61640 10 monclient: get_monmap_and_config 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.694+0000 7f5f88a61640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.694+0000 7f5f88a61640 10 monclient: monmap: 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:52.695391+0000 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:52.695391+0000 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.695+0000 7f5f88a61640 1 Processor -- start 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.695+0000 7f5f88a61640 1 -- start start 2026-02-22T21:42:52.696 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.695+0000 7f5f88a61640 10 monclient: init 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient: _add_conns ranks=[5,2,4,0,3,1] 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f5f841528b0 0x7f5f840c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-c con 0x7f5f841528b0 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f840cd9f0 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f84073070 0x7f5f84073a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-b con 0x7f5f84073070 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f840c9220 con 0x7f5f840cd9f0 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f5f840cd560 con 0x7f5f84073070 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f5f840cc670 con 0x7f5f841528b0 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.697 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f88a61640 10 monclient(hunting): authenticate will time out at 1448.276087s 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82575640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f5f841528b0 0x7f5f840c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f84073070 0x7f5f84073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f5f84073070 0x7f5f84073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:41244/0 (socket says 10.20.193.51:41244) 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 1 -- 10.20.193.51:0/134840522 learned_addr learned my addr 10.20.193.51:0/134840522 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 10 monclient(hunting): get_auth_request con 0x7f5f84073070 auth_method 0 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82575640 10 monclient(hunting): get_auth_request con 0x7f5f841528b0 auth_method 0 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.696+0000 7f5f82575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.698 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.697+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 10 monclient(hunting): handle_auth_done global_id 5314 payload 274 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 -- 10.20.193.51:0/134840522 >> v2:10.20.193.125:3300/0 conn(0x7f5f841528b0 msgr2=0x7f5f840c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 --2- 10.20.193.51:0/134840522 >> v2:10.20.193.125:3300/0 conn(0x7f5f841528b0 0x7f5f840c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 -- 10.20.193.51:0/134840522 >> v1:10.20.193.125:6789/0 conn(0x7f5f840cd9f0 legacy=0x7f5f8415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 monclient: found mon.noname-b 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.699 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 -- 10.20.193.51:0/134840522 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f840cc950 con 0x7f5f84073070 2026-02-22T21:42:52.700 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.698+0000 7f5f82d76640 1 --2- 10.20.193.51:0/134840522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f84073070 0x7f5f84073a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5f840cf130 tx=0x7f5f7402f270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3411902d89dbae5c server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 1 -- 10.20.193.51:0/134840522 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7402fd70 con 0x7f5f84073070 2026-02-22T21:42:52.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.701 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: dump: 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 1 -- 10.20.193.51:0/134840522 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f740373f0 con 0x7f5f84073070 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 1 -- 10.20.193.51:0/134840522 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740377c0 con 0x7f5f84073070 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f88a61640 5 monclient: authenticate success, global_id 5314 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f88a61640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f88a61640 4 set_mon_vals no callback set 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.700+0000 7f5f81573640 10 monclient: dump: 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.702 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 10 monclient: shutdown 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- 10.20.193.51:0/134840522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f84073070 msgr2=0x7f5f84073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 --2- 10.20.193.51:0/134840522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f84073070 0x7f5f84073a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5f840cf130 tx=0x7f5f7402f270 comp rx=0 tx=0).stop 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- 10.20.193.51:0/134840522 shutdown_connections 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 --2- 10.20.193.51:0/134840522 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f84073070 0x7f5f84073a40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 --2- 10.20.193.51:0/134840522 >> v2:10.20.193.125:3300/0 conn(0x7f5f841528b0 0x7f5f840c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- 10.20.193.51:0/134840522 >> 10.20.193.51:0/134840522 conn(0x7f5f840ce360 msgr2=0x7f5f840ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.703 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- 10.20.193.51:0/134840522 shutdown_connections 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- 10.20.193.51:0/134840522 wait complete. 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 10 monclient: monmap: 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 librados: starting msgr at 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 librados: starting objecter 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 Processor -- start 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 -- start start 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 librados: setting wanted keys 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 1 librados: calling monclient init 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.701+0000 7f5f88a61640 10 monclient: init 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient: _add_conns ranks=[2,1,0] 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f84073070 0x7f5f840c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-c con 0x7f5f84073070 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 0x7f5f84149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-b con 0x7f5f840cd9f0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f841528b0 0x7f5f840c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): picked mon.noname-a con 0x7f5f841528b0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f5f840cd1f0 con 0x7f5f841528b0 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f5f840ccf10 con 0x7f5f840cd9f0 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84152c90 con 0x7f5f84073070 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f88a61640 10 monclient(hunting): authenticate will time out at 1448.282217s 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82575640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f84073070 0x7f5f840c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.704 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82d76640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f841528b0 0x7f5f840c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 0x7f5f84149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 0x7f5f84149bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:41250/0 (socket says 10.20.193.51:41250) 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 1 -- 10.20.193.51:0/3252800461 learned_addr learned my addr 10.20.193.51:0/3252800461 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 10 monclient(hunting): get_auth_request con 0x7f5f840cd9f0 auth_method 0 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f81d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82d76640 10 monclient(hunting): get_auth_request con 0x7f5f841528b0 auth_method 0 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.705 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82575640 10 monclient(hunting): get_auth_request con 0x7f5f84073070 auth_method 0 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.702+0000 7f5f82575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f81d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f81d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f81d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.703+0000 7f5f82575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 10 monclient(hunting): handle_auth_done global_id 5317 payload 824 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 -- 10.20.193.51:0/3252800461 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f84073070 msgr2=0x7f5f840c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f5f84073070 0x7f5f840c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 -- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f841528b0 msgr2=0x7f5f840c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f5f841528b0 0x7f5f840c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 monclient: found mon.noname-b 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.706 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.704+0000 7f5f81d74640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8414a580 con 0x7f5f840cd9f0 2026-02-22T21:42:52.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.705+0000 7f5f81d74640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 0x7f5f84149bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00c520 tx=0x7f5f6c00c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.705+0000 7f5f6affd640 10 client.?.objecter ms_handle_connect 0x7f5f840cd9f0 2026-02-22T21:42:52.707 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.705+0000 7f5f6affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:52.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0052d0 con 0x7f5f840cd9f0 2026-02-22T21:42:52.708 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: dump: 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 5 monclient: authenticate success, global_id 5317 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 monclient: _renew_subs 2026-02-22T21:42:52.709 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8414a860 con 0x7f5f840cd9f0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 client.5317.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 monclient: _renew_subs 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f840c5a40 con 0x7f5f840cd9f0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 1 librados: init done 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f6c005470 con 0x7f5f840cd9f0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f80d72640 4 set_mon_vals no callback set 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0038f0 con 0x7f5f840cd9f0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f6affd640 10 monclient: dump: 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.710 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 client.5317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 15 client.5317.objecter _session_command_op_assign -1 1 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 20 client.5317.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 client.5317.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:52.711 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.706+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f8414ad80 con 0x7f5f840cd9f0 2026-02-22T21:42:52.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.717+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f5f6c003a90 con 0x7f5f840cd9f0 2026-02-22T21:42:52.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.718+0000 7f5f6affd640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5f50078e00 0x7f5f5007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.718+0000 7f5f82575640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5f50078e00 0x7f5f5007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.719 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.718+0000 7f5f82575640 10 monclient: get_auth_request con 0x7f5f50078e00 auth_method 0 2026-02-22T21:42:52.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.718+0000 7f5f82575640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5f50078e00 0x7f5f5007b2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009ba0 tx=0x7f5f78009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.720 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.718+0000 7f5f6affd640 10 client.5317.objecter ms_handle_connect 0x7f5f50078e00 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f5f6c0979b0 con 0x7f5f840cd9f0 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 10 client.5317.objecter ms_dispatch2 0x7f5f84000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 3 client.5317.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 3 client.5317.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 10 client.5317.objecter checking command 1 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f5f500812e0 0x7f5f50083700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter _get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 15 client.5317.objecter _session_command_op_remove -1 1 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter _get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 15 client.5317.objecter _session_command_op_assign 7 1 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 4 2026-02-22T21:42:52.721 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 10 client.5317.objecter _send_command 1 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f50083dd0 con 0x7f5f500812e0 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 20 client.5317.objecter dump_active .. 0 homeless 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5f6c05e360 con 0x7f5f840cd9f0 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f6affd640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f82d76640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f5f500812e0 0x7f5f50083700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.722 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.720+0000 7f5f82d76640 10 monclient: get_auth_request con 0x7f5f500812e0 auth_method 0 2026-02-22T21:42:52.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.722+0000 7f5f82d76640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f5f500812e0 0x7f5f50083700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.723 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.722+0000 7f5f6affd640 10 client.5317.objecter ms_handle_connect 0x7f5f500812e0 2026-02-22T21:42:52.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.731+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== osd.7 v2:10.20.193.140:6808/1140578699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f5f50083dd0 con 0x7f5f500812e0 2026-02-22T21:42:52.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.731+0000 7f5f6affd640 10 client.5317.objecter ms_dispatch2 0x7f5f84000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:52.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.731+0000 7f5f6affd640 10 client.5317.objecter _finish_command 1 = system:0 2026-02-22T21:42:52.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.731+0000 7f5f6affd640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 4 2026-02-22T21:42:52.733 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.731+0000 7f5f6affd640 15 client.5317.objecter _session_command_op_remove 7 1 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 10 client.5317.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 15 client.5317.objecter _session_command_op_assign -1 2 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 20 client.5317.objecter _get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 20 client.5317.objecter _get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 15 client.5317.objecter _session_command_op_remove -1 2 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 20 client.5317.objecter get_session s=0x7f5f50080c10 osd=7 3 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 15 client.5317.objecter _session_command_op_assign 7 2 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 4 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 10 client.5317.objecter _send_command 2 2026-02-22T21:42:52.739 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.737+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 --> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5f840cc260 con 0x7f5f500812e0 2026-02-22T21:42:52.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f6affd640 1 -- 10.20.193.51:0/3252800461 <== osd.7 v2:10.20.193.140:6808/1140578699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5f840cc260 con 0x7f5f500812e0 2026-02-22T21:42:52.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f6affd640 10 client.5317.objecter ms_dispatch2 0x7f5f84000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f6affd640 10 client.5317.objecter _finish_command 2 = system:0 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f6affd640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 4 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f6affd640 15 client.5317.objecter _session_command_op_remove 7 2 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:52.746 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 10 librados: watch_flush enter 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 10 librados: watch_flush exit 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 10 client.5317.objecter close_session for osd.7 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f5f500812e0 msgr2=0x7f5f50083700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.140:6808/1140578699,v1:10.20.193.140:6809/1140578699] conn(0x7f5f500812e0 0x7f5f50083700 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 20 client.5317.objecter put_session s=0x7f5f50080c10 osd=7 1 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 20 client.5317.objecter shutdown clearing up homeless session... 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 10 client.5317.objecter successfully canceled tick 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5f50078e00 msgr2=0x7f5f5007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f5f50078e00 0x7f5f5007b2c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009ba0 tx=0x7f5f78009290 comp rx=0 tx=0).stop 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 10 monclient: shutdown 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 msgr2=0x7f5f84149bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 --2- 10.20.193.51:0/3252800461 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f5f840cd9f0 0x7f5f84149bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00c520 tx=0x7f5f6c00c9f0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f82d76640 1 -- 10.20.193.51:0/3252800461 reap_dead start 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 shutdown_connections 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 >> 10.20.193.51:0/3252800461 conn(0x7f5f840ce360 msgr2=0x7f5f8414dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 shutdown_connections 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 -- 10.20.193.51:0/3252800461 wait complete. 2026-02-22T21:42:52.747 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.745+0000 7f5f88a61640 1 librados: shutdown 2026-02-22T21:42:52.753 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 10 monclient: get_monmap_and_config 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 10 monclient: monmap: 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:52.820825+0000 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:52.820825+0000 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:52.821 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 1 Processor -- start 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 1 -- start start 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.820+0000 7f6ceaca6640 10 monclient: init 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient: _add_conns ranks=[3,2,5,4,0,1] 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6ce4150220 0x7f6ce40c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ce4150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:52.822 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ce4150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f6ce415c710 0x7f6ce406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ce415c710 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ce415a920 con 0x7f6ce4150600 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f6ce415af60 con 0x7f6ce4150220 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f6ce415b240 con 0x7f6ce415c710 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ceaca6640 10 monclient(hunting): authenticate will time out at 1448.401227s 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6ce4150220 0x7f6ce40c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.823 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f6ce4150220 0x7f6ce40c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35670/0 (socket says 10.20.193.51:35670) 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/1225112235 learned_addr learned my addr 10.20.193.51:0/1225112235 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 10 monclient(hunting): get_auth_request con 0x7f6ce4150220 auth_method 0 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce8a1b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.821+0000 7f6ce921c640 1 --2- 10.20.193.51:0/1225112235 >> v2:10.20.193.125:3300/0 conn(0x7f6ce415c710 0x7f6ce406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): get_auth_request con 0x7f6ce415c710 auth_method 0 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.824 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_done global_id 5304 payload 274 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/1225112235 >> v2:10.20.193.125:3300/0 conn(0x7f6ce415c710 msgr2=0x7f6ce406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 1 --2- 10.20.193.51:0/1225112235 >> v2:10.20.193.125:3300/0 conn(0x7f6ce415c710 0x7f6ce406fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.822+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/1225112235 >> v1:10.20.193.125:6789/0 conn(0x7f6ce4150600 legacy=0x7f6ce415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 1 monclient: found mon.noname-a 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/1225112235 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce415b520 con 0x7f6ce4150220 2026-02-22T21:42:52.825 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.823+0000 7f6ce8a1b640 1 --2- 10.20.193.51:0/1225112235 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce40c73a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40c65c0 tx=0x7f6cd402ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c392904c885842d server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 1 -- 10.20.193.51:0/1225112235 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd403c070 con 0x7f6ce4150220 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 10 monclient: dump: 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.826 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6ceaca6640 5 monclient: authenticate success, global_id 5304 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.824+0000 7f6ceaca6640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 1 -- 10.20.193.51:0/1225112235 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd402fc70 con 0x7f6ce4150220 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 1 -- 10.20.193.51:0/1225112235 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4037660 con 0x7f6ce4150220 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:52.827 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 4 set_mon_vals no callback set 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6cdb7fe640 10 monclient: dump: 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 10 monclient: shutdown 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 1 -- 10.20.193.51:0/1225112235 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 msgr2=0x7f6ce40c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/1225112235 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce40c73a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40c65c0 tx=0x7f6cd402ef60 comp rx=0 tx=0).stop 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 1 -- 10.20.193.51:0/1225112235 shutdown_connections 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/1225112235 >> v2:10.20.193.125:3300/0 conn(0x7f6ce415c710 0x7f6ce406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.825+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/1225112235 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce40c73a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 -- 10.20.193.51:0/1225112235 >> 10.20.193.51:0/1225112235 conn(0x7f6ce40cd7b0 msgr2=0x7f6ce40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 -- 10.20.193.51:0/1225112235 shutdown_connections 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 -- 10.20.193.51:0/1225112235 wait complete. 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 10 monclient: build_initial_monmap 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 10 monclient: monmap: 2026-02-22T21:42:52.828 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:52.829 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 librados: starting msgr at 2026-02-22T21:42:52.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 librados: starting objecter 2026-02-22T21:42:52.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 Processor -- start 2026-02-22T21:42:52.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 -- start start 2026-02-22T21:42:52.830 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 librados: setting wanted keys 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 1 librados: calling monclient init 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.826+0000 7f6ceaca6640 10 monclient: init 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ce4150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6ce4150600 0x7f6ce40c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ce4150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6ce415c710 0x7f6ce41e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ce415c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce40701e0 con 0x7f6ce4150220 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce415bae0 con 0x7f6ce415c710 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4070360 con 0x7f6ce4150600 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ceaca6640 10 monclient(hunting): authenticate will time out at 1448.407480s 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce921c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6ce415c710 0x7f6ce41e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35684/0 (socket says 10.20.193.51:35684) 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/2640080060 learned_addr learned my addr 10.20.193.51:0/2640080060 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 10 monclient(hunting): get_auth_request con 0x7f6ce4150220 auth_method 0 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce8a1b640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.831 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce921c640 10 monclient(hunting): get_auth_request con 0x7f6ce415c710 auth_method 0 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce921c640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.827+0000 7f6ce921c640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6ce921c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6ce4150600 0x7f6ce40c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): get_auth_request con 0x7f6ce4150600 auth_method 0 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.828+0000 7f6cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 10 monclient(hunting): handle_auth_done global_id 5307 payload 824 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/2640080060 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6ce4150600 msgr2=0x7f6ce40c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6ce4150600 0x7f6ce40c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/2640080060 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6ce415c710 msgr2=0x7f6ce41e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6ce415c710 0x7f6ce41e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 10 monclient: _finish_hunting 0 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 monclient: found mon.noname-a 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:52.832 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.829+0000 7f6ce8a1b640 1 -- 10.20.193.51:0/2640080060 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce41f2760 con 0x7f6ce4150220 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.830+0000 7f6ce8a1b640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009a70 tx=0x7f6cd40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.830+0000 7f6cd97fa640 10 client.?.objecter ms_handle_connect 0x7f6ce4150220 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.830+0000 7f6cd97fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd403c040 con 0x7f6ce4150220 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: dump: 2026-02-22T21:42:52.833 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd4042630 con 0x7f6ce4150220 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4042a20 con 0x7f6ce4150220 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cdaffd640 4 set_mon_vals no callback set 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6cd97fa640 10 monclient: dump: 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:52.834 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 5 monclient: authenticate success, global_id 5307 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _renew_subs 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce40c1ad0 con 0x7f6ce4150220 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 client.5307.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _renew_subs 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce40c2090 con 0x7f6ce4150220 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 1 librados: init done 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:52.835 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.832+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce406fca0 con 0x7f6ce4150220 2026-02-22T21:42:52.844 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.843+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f6cd4042bc0 con 0x7f6ce4150220 2026-02-22T21:42:52.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.844+0000 7f6cd97fa640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 0x7f6cb407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:52.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.844+0000 7f6cdbfff640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 0x7f6cb407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:52.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.844+0000 7f6cdbfff640 10 monclient: get_auth_request con 0x7f6cb4078e00 auth_method 0 2026-02-22T21:42:52.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.844+0000 7f6cd97fa640 10 client.5307.objecter ms_handle_connect 0x7f6cb4078e00 2026-02-22T21:42:52.845 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.844+0000 7f6cdbfff640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 0x7f6cb407b2c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4154040 tx=0x7f6ccc007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:52.846 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.845+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f6cd402fb30 con 0x7f6ce4150220 2026-02-22T21:42:52.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.845+0000 7f6cd97fa640 10 client.5307.objecter ms_dispatch2 0x7f6ce4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:52.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.845+0000 7f6cd97fa640 3 client.5307.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:52.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.845+0000 7f6cd97fa640 3 client.5307.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:52.847 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.845+0000 7f6cd97fa640 20 client.5307.objecter dump_active .. 0 homeless 2026-02-22T21:42:52.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.945+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f6ce41f2760 con 0x7f6ce4150220 2026-02-22T21:42:52.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.945+0000 7f6cd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:52.947 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:52.945+0000 7f6cd97fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:53.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.031+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f6ce4157f60 con 0x7f6cb4078e00 2026-02-22T21:42:53.033 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.031+0000 7f6cd97fa640 1 -- 10.20.193.51:0/2640080060 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6ce4157f60 con 0x7f6cb4078e00 2026-02-22T21:42:53.034 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 7 to deep-scrub 2026-02-22T21:42:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 10 librados: watch_flush enter 2026-02-22T21:42:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 10 librados: watch_flush exit 2026-02-22T21:42:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 20 client.5307.objecter shutdown clearing up homeless session... 2026-02-22T21:42:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 10 client.5307.objecter successfully canceled tick 2026-02-22T21:42:53.035 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 msgr2=0x7f6cb407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 0x7f6cb407b2c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4154040 tx=0x7f6ccc007f30 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 10 monclient: shutdown 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 msgr2=0x7f6ce41f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009a70 tx=0x7f6cd40047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 shutdown_connections 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f6cb4078e00 0x7f6cb407b2c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f6ce415c710 0x7f6ce41e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f6ce4150600 0x7f6ce40c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 --2- 10.20.193.51:0/2640080060 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f6ce4150220 0x7f6ce41f2220 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.036 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 >> 10.20.193.51:0/2640080060 conn(0x7f6ce40cd7b0 msgr2=0x7f6ce40cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 shutdown_connections 2026-02-22T21:42:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 -- 10.20.193.51:0/2640080060 wait complete. 2026-02-22T21:42:53.037 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.032+0000 7f6ceaca6640 1 librados: shutdown 2026-02-22T21:42:53.042 INFO:tasks.ceph:Scrubbing osd.11 2026-02-22T21:42:53.042 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:53.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.111+0000 7f018f64c640 10 monclient: get_monmap_and_config 2026-02-22T21:42:53.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.111+0000 7f018f64c640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.113 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.111+0000 7f018f64c640 10 monclient: monmap: 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:53.112506+0000 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:53.112506+0000 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:53.114 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:53.115 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:53.115 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:53.115 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 Processor -- start 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 -- start start 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient: init 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient: _add_conns ranks=[1,0,4,5,3,2] 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f018814e810 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0188156750 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:53.118 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f018814ebf0 0x7f0188158a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-b con 0x7f018814ebf0 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018815a330 con 0x7f0188156750 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018815a610 con 0x7f018814e810 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f018815a8f0 con 0x7f018814ebf0 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.112+0000 7f018f64c640 10 monclient(hunting): authenticate will time out at 1448.692977s 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018cbc0640 1 --1- >> v1:10.20.193.51:6789/0 conn(0x7f0188156750 0x7f01880c8500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.51:6789/0 says I am v1:10.20.193.51:59036/0 (socket says 10.20.193.51:59036) 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018cbc0640 1 -- 10.20.193.51:0/618478464 learned_addr learned my addr 10.20.193.51:0/618478464 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018dbc2640 1 --2- 10.20.193.51:0/618478464 >> v2:10.20.193.68:3300/0 conn(0x7f018814ebf0 0x7f0188158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018dbc2640 10 monclient(hunting): get_auth_request con 0x7f018814ebf0 auth_method 0 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018dbc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.119 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018dbc2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.113+0000 7f018dbc2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 1 -- 10.20.193.51:0/618478464 <== mon.0 v1:10.20.193.51:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178183947 0 0) 0x7f018815a330 con 0x7f0188156750 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 10 monclient(hunting): _negotiate 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 10 monclient(hunting): my global_id is 5310 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.114+0000 7f0177fff640 1 -- 10.20.193.51:0/618478464 --> v1:10.20.193.51:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0170003620 con 0x7f0188156750 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 10 monclient(hunting): handle_auth_done global_id 5323 payload 274 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 1 -- 10.20.193.51:0/618478464 >> v1:10.20.193.68:6789/0 conn(0x7f018814e810 legacy=0x7f0188156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.120 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 1 -- 10.20.193.51:0/618478464 >> v1:10.20.193.51:6789/0 conn(0x7f0188156750 legacy=0x7f01880c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 1 monclient: found mon.noname-b 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.115+0000 7f018dbc2640 1 -- 10.20.193.51:0/618478464 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018815abd0 con 0x7f018814ebf0 2026-02-22T21:42:53.121 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.116+0000 7f018dbc2640 1 --2- 10.20.193.51:0/618478464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188158a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0184009d90 tx=0x7f018402f230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=491b8974394d691e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 1 -- 10.20.193.51:0/618478464 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018403c070 con 0x7f018814ebf0 2026-02-22T21:42:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.127 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: dump: 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.128 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 1 -- 10.20.193.51:0/618478464 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184037440 con 0x7f018814ebf0 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 5 monclient: authenticate success, global_id 5323 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 1 -- 10.20.193.51:0/618478464 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184037810 con 0x7f018814ebf0 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 4 set_mon_vals no callback set 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.129 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f0177fff640 10 monclient: dump: 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 10 monclient: shutdown 2026-02-22T21:42:53.130 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 -- 10.20.193.51:0/618478464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 msgr2=0x7f0188158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 --2- 10.20.193.51:0/618478464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188158a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0184009d90 tx=0x7f018402f230 comp rx=0 tx=0).stop 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 -- 10.20.193.51:0/618478464 shutdown_connections 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 --2- 10.20.193.51:0/618478464 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188158a70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 -- 10.20.193.51:0/618478464 >> 10.20.193.51:0/618478464 conn(0x7f01880c9d50 msgr2=0x7f01880ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.118+0000 7f018f64c640 1 -- 10.20.193.51:0/618478464 shutdown_connections 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 -- 10.20.193.51:0/618478464 wait complete. 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 10 monclient: monmap: 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:53.131 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 librados: starting msgr at 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 librados: starting objecter 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 Processor -- start 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 -- start start 2026-02-22T21:42:53.132 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 librados: setting wanted keys 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 1 librados: calling monclient init 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.119+0000 7f018f64c640 10 monclient: init 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 0x7f018806b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-a con 0x7f018814e810 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-b con 0x7f018814ebf0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0188156750 0x7f01881f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): picked mon.noname-c con 0x7f0188156750 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f01880c8c10 con 0x7f018814e810 2026-02-22T21:42:53.133 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f018815b190 con 0x7f018814ebf0 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f01880c8d90 con 0x7f0188156750 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018f64c640 10 monclient(hunting): authenticate will time out at 1448.700330s 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 0x7f018806b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 0x7f018806b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35686/0 (socket says 10.20.193.51:35686) 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 1 -- 10.20.193.51:0/2659744555 learned_addr learned my addr 10.20.193.51:0/2659744555 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): get_auth_request con 0x7f018814e810 auth_method 0 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.134 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018cbc0640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018dbc2640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0188156750 0x7f01881f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018cbc0640 10 monclient(hunting): get_auth_request con 0x7f018814ebf0 auth_method 0 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018cbc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018cbc0640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018cbc0640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.120+0000 7f018d3c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): get_auth_request con 0x7f0188156750 auth_method 0 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.135 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018cbc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018cbc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018cbc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.121+0000 7f018dbc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 10 monclient(hunting): handle_auth_done global_id 5313 payload 824 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 -- 10.20.193.51:0/2659744555 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0188156750 msgr2=0x7f01881f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f0188156750 0x7f01881f2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 -- 10.20.193.51:0/2659744555 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 msgr2=0x7f0188152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f018814ebf0 0x7f0188152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.136 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.122+0000 7f018d3c1640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01881554b0 con 0x7f018814e810 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.123+0000 7f018d3c1640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 0x7f018806b890 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f017800c700 tx=0x7f017800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.123+0000 7f0175ffb640 10 client.?.objecter ms_handle_connect 0x7f018814e810 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.123+0000 7f0175ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01780052d0 con 0x7f018814e810 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: dump: 2026-02-22T21:42:53.137 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.138 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0178005470 con 0x7f018814e810 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 5 monclient: authenticate success, global_id 5313 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 monclient: _renew_subs 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0188153390 con 0x7f018814e810 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017800a860 con 0x7f018814e810 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f01777fe640 4 set_mon_vals no callback set 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f0175ffb640 10 monclient: dump: 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.139 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 client.5313.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 monclient: _renew_subs 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0188153950 con 0x7f018814e810 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 1 librados: init done 2026-02-22T21:42:53.140 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 client.5313.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 15 client.5313.objecter _session_command_op_assign -1 1 2026-02-22T21:42:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 20 client.5313.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 client.5313.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.141 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.125+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0188158a70 con 0x7f018814e810 2026-02-22T21:42:53.143 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.136+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f017800aa00 con 0x7f018814e810 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.137+0000 7f0175ffb640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0158078e80 0x7f015807b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.137+0000 7f018cbc0640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0158078e80 0x7f015807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.137+0000 7f018cbc0640 10 monclient: get_auth_request con 0x7f0158078e80 auth_method 0 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.137+0000 7f0175ffb640 10 client.5313.objecter ms_handle_connect 0x7f0158078e80 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.137+0000 7f018cbc0640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0158078e80 0x7f015807b340 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f01880c77b0 tx=0x7f017c007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f017800ad20 con 0x7f018814e810 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 10 client.5313.objecter ms_dispatch2 0x7f0188000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 3 client.5313.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 3 client.5313.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 10 client.5313.objecter checking command 1 2026-02-22T21:42:53.144 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f01580813c0 0x7f01580837e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter _get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 15 client.5313.objecter _session_command_op_remove -1 1 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter _get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 15 client.5313.objecter _session_command_op_assign 11 1 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 4 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 10 client.5313.objecter _send_command 1 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0178013e70 con 0x7f01580813c0 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 20 client.5313.objecter dump_active .. 0 homeless 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f017805e290 con 0x7f018814e810 2026-02-22T21:42:53.145 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f0175ffb640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.138+0000 7f018dbc2640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f01580813c0 0x7f01580837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.139+0000 7f018dbc2640 10 monclient: get_auth_request con 0x7f01580813c0 auth_method 0 2026-02-22T21:42:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.140+0000 7f018dbc2640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f01580813c0 0x7f01580837e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.146 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.140+0000 7f0175ffb640 10 client.5313.objecter ms_handle_connect 0x7f01580813c0 2026-02-22T21:42:53.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.151+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== osd.11 v2:10.20.193.140:6816/2298497951 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7f0178013e70 con 0x7f01580813c0 2026-02-22T21:42:53.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.151+0000 7f0175ffb640 10 client.5313.objecter ms_dispatch2 0x7f0188000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:53.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.151+0000 7f0175ffb640 10 client.5313.objecter _finish_command 1 = system:0 2026-02-22T21:42:53.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.151+0000 7f0175ffb640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 4 2026-02-22T21:42:53.153 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.151+0000 7f0175ffb640 15 client.5313.objecter _session_command_op_remove 11 1 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 10 client.5313.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 15 client.5313.objecter _session_command_op_assign -1 2 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 20 client.5313.objecter _get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 20 client.5313.objecter _get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.160 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 15 client.5313.objecter _session_command_op_remove -1 2 2026-02-22T21:42:53.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 20 client.5313.objecter get_session s=0x7f0158080cf0 osd=11 3 2026-02-22T21:42:53.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 15 client.5313.objecter _session_command_op_assign 11 2 2026-02-22T21:42:53.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 4 2026-02-22T21:42:53.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 10 client.5313.objecter _send_command 2 2026-02-22T21:42:53.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.158+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 --> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0188157520 con 0x7f01580813c0 2026-02-22T21:42:53.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f0175ffb640 1 -- 10.20.193.51:0/2659744555 <== osd.11 v2:10.20.193.140:6816/2298497951 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0188157520 con 0x7f01580813c0 2026-02-22T21:42:53.166 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f0175ffb640 10 client.5313.objecter ms_dispatch2 0x7f0188000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:53.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f0175ffb640 10 client.5313.objecter _finish_command 2 = system:0 2026-02-22T21:42:53.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f0175ffb640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 4 2026-02-22T21:42:53.167 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f0175ffb640 15 client.5313.objecter _session_command_op_remove 11 2 2026-02-22T21:42:53.167 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:53.168 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:53.168 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:53.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 10 librados: watch_flush enter 2026-02-22T21:42:53.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 10 librados: watch_flush exit 2026-02-22T21:42:53.170 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 10 client.5313.objecter close_session for osd.11 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f01580813c0 msgr2=0x7f01580837e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.140:6816/2298497951,v1:10.20.193.140:6817/2298497951] conn(0x7f01580813c0 0x7f01580837e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 20 client.5313.objecter put_session s=0x7f0158080cf0 osd=11 1 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 20 client.5313.objecter shutdown clearing up homeless session... 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 10 client.5313.objecter successfully canceled tick 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0158078e80 msgr2=0x7f015807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.171 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f0158078e80 0x7f015807b340 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f01880c77b0 tx=0x7f017c007dd0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 10 monclient: shutdown 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 msgr2=0x7f018806b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 --2- 10.20.193.51:0/2659744555 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f018814e810 0x7f018806b890 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f017800c700 tx=0x7f017800cbd0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018dbc2640 1 -- 10.20.193.51:0/2659744555 reap_dead start 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 shutdown_connections 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 >> 10.20.193.51:0/2659744555 conn(0x7f01880c9d50 msgr2=0x7f018814c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.172 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 shutdown_connections 2026-02-22T21:42:53.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 -- 10.20.193.51:0/2659744555 wait complete. 2026-02-22T21:42:53.173 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.165+0000 7f018f64c640 1 librados: shutdown 2026-02-22T21:42:53.175 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 10 monclient: get_monmap_and_config 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 10 monclient: monmap: 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.245 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:53.242075+0000 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:53.242075+0000 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 1 Processor -- start 2026-02-22T21:42:53.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 1 -- start start 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.241+0000 7f75cf20b640 10 monclient: init 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient: _add_conns ranks=[5,2,4,1,3,0] 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 0x7f75c80c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-c con 0x7f75c8150220 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75c8150600 addr v1:10.20.193.125:6789/0 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f75c815c710 0x7f75c806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-b con 0x7f75c815c710 addr v2:10.20.193.68:3300/0 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 1 -- --> v1:10.20.193.125:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c815a920 con 0x7f75c8150600 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 1 -- --> v2:10.20.193.68:3300/0 -- mon_getmap magic: 0 -- 0x7f75c815af60 con 0x7f75c815c710 2026-02-22T21:42:53.247 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7f75c815b240 con 0x7f75c8150220 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cf20b640 10 monclient(hunting): authenticate will time out at 1448.822496s 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75ccf80640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 0x7f75c80c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.242+0000 7f75cd781640 1 --2- >> v2:10.20.193.68:3300/0 conn(0x7f75c815c710 0x7f75c806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 0x7f75c80c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:37038/0 (socket says 10.20.193.51:37038) 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 1 -- 10.20.193.51:0/3291118838 learned_addr learned my addr 10.20.193.51:0/3291118838 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): get_auth_request con 0x7f75c8150220 auth_method 0 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.248 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): get_auth_request con 0x7f75c815c710 auth_method 0 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.249 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.243+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 10 monclient(hunting): handle_auth_done global_id 5329 payload 274 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 1 -- 10.20.193.51:0/3291118838 >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 msgr2=0x7f75c80c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 1 --2- 10.20.193.51:0/3291118838 >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 0x7f75c80c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 1 -- 10.20.193.51:0/3291118838 >> v1:10.20.193.125:6789/0 conn(0x7f75c8150600 legacy=0x7f75c815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 1 monclient: found mon.noname-b 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.244+0000 7f75cd781640 1 -- 10.20.193.51:0/3291118838 --> v2:10.20.193.68:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c815b520 con 0x7f75c815c710 2026-02-22T21:42:53.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.245+0000 7f75cd781640 1 --2- 10.20.193.51:0/3291118838 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c806fca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f75c8070940 tx=0x7f75c402f250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c04482b7a2b50540 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 1 -- 10.20.193.51:0/3291118838 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c403c070 con 0x7f75c815c710 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 10 monclient: dump: 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.258 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75cf20b640 5 monclient: authenticate success, global_id 5329 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.246+0000 7f75cf20b640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 1 -- 10.20.193.51:0/3291118838 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c4037440 con 0x7f75c815c710 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 1 -- 10.20.193.51:0/3291118838 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4037810 con 0x7f75c815c710 2026-02-22T21:42:53.259 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 4 set_mon_vals no callback set 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75bf7fe640 10 monclient: dump: 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.260 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 10 monclient: shutdown 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 -- 10.20.193.51:0/3291118838 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 msgr2=0x7f75c806fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 --2- 10.20.193.51:0/3291118838 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c806fca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f75c8070940 tx=0x7f75c402f250 comp rx=0 tx=0).stop 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 -- 10.20.193.51:0/3291118838 shutdown_connections 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 --2- 10.20.193.51:0/3291118838 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c806fca0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 --2- 10.20.193.51:0/3291118838 >> v2:10.20.193.125:3300/0 conn(0x7f75c8150220 0x7f75c80c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 -- 10.20.193.51:0/3291118838 >> 10.20.193.51:0/3291118838 conn(0x7f75c80cd7b0 msgr2=0x7f75c80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 -- 10.20.193.51:0/3291118838 shutdown_connections 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 -- 10.20.193.51:0/3291118838 wait complete. 2026-02-22T21:42:53.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 10 monclient: monmap: 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 librados: starting msgr at 2026-02-22T21:42:53.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 librados: starting objecter 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.247+0000 7f75cf20b640 1 Processor -- start 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 -- start start 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 librados: setting wanted keys 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 librados: calling monclient init 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient: init 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient: _add_conns ranks=[0,2,1] 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f75c8150220 0x7f75c8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-a con 0x7f75c8150220 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f75c8150600 0x7f75c814ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-c con 0x7f75c8150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): picked mon.noname-b con 0x7f75c815c710 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f75c80701e0 con 0x7f75c8150220 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f75c815bae0 con 0x7f75c815c710 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f75c8070360 con 0x7f75c8150600 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cf20b640 10 monclient(hunting): authenticate will time out at 1448.828495s 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75cd781640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75ccf80640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f75c8150220 0x7f75c8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.248+0000 7f75bffff640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f75c8150600 0x7f75c814ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.68:3300/0 says I am v2:10.20.193.51:41296/0 (socket says 10.20.193.51:41296) 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 1 -- 10.20.193.51:0/4036084301 learned_addr learned my addr 10.20.193.51:0/4036084301 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): get_auth_request con 0x7f75c815c710 auth_method 0 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): get_auth_request con 0x7f75c8150220 auth_method 0 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): get_auth_request con 0x7f75c8150600 auth_method 0 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75cd781640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75ccf80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.249+0000 7f75bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 10 monclient(hunting): handle_auth_done global_id 5332 payload 824 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 -- 10.20.193.51:0/4036084301 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f75c8150600 msgr2=0x7f75c814ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f75c8150600 0x7f75c814ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 -- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f75c8150220 msgr2=0x7f75c8158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f75c8150220 0x7f75c8158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 monclient: found mon.noname-b 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 10 monclient: _send_mon_message to mon.noname-b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.250+0000 7f75cd781640 1 -- 10.20.193.51:0/4036084301 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c8159be0 con 0x7f75c815c710 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.251+0000 7f75cd781640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f75c402f780 tx=0x7f75c4004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.251+0000 7f75bd7fa640 10 client.?.objecter ms_handle_connect 0x7f75c815c710 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.251+0000 7f75bd7fa640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c403c050 con 0x7f75c815c710 2026-02-22T21:42:53.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 10 monclient: dump: 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.268 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 5 monclient: authenticate success, global_id 5332 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 10 monclient: _renew_subs 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c8159340 con 0x7f75c815c710 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 10 client.5332.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 10 monclient: _renew_subs 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c8159900 con 0x7f75c815c710 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.252+0000 7f75cf20b640 1 librados: init done 2026-02-22T21:42:53.269 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c4048070 con 0x7f75c815c710 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c4035a50 con 0x7f75c815c710 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75beffd640 4 set_mon_vals no callback set 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75bd7fa640 10 monclient: dump: 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.270 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75cf20b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75cf20b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75cf20b640 10 monclient: _send_mon_message to mon.b at v2:10.20.193.68:3300/0 2026-02-22T21:42:53.271 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.253+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c80cce20 con 0x7f75c815c710 2026-02-22T21:42:53.359 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.357+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f75c4035bf0 con 0x7f75c815c710 2026-02-22T21:42:53.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.358+0000 7f75bd7fa640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 0x7f759807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.358+0000 7f75ccf80640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 0x7f759807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.358+0000 7f75ccf80640 10 monclient: get_auth_request con 0x7f7598078e00 auth_method 0 2026-02-22T21:42:53.360 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.358+0000 7f75ccf80640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 0x7f759807b2c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006dc0 tx=0x7f75b8009340 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.358+0000 7f75bd7fa640 10 client.5332.objecter ms_handle_connect 0x7f7598078e00 2026-02-22T21:42:53.361 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.360+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f75c40845e0 con 0x7f75c815c710 2026-02-22T21:42:53.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.360+0000 7f75bd7fa640 10 client.5332.objecter ms_dispatch2 0x7f75c8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:53.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.360+0000 7f75bd7fa640 3 client.5332.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:53.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.360+0000 7f75bd7fa640 3 client.5332.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:53.362 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.360+0000 7f75bd7fa640 20 client.5332.objecter dump_active .. 0 homeless 2026-02-22T21:42:53.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.366+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mon.1 v2:10.20.193.68:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f75c40bde20 con 0x7f75c815c710 2026-02-22T21:42:53.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.366+0000 7f75bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.366+0000 7f75bd7fa640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:53.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.449+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "11", "target": ["mon-mgr", ""]}) -- 0x7f75c80ce0c0 con 0x7f7598078e00 2026-02-22T21:42:53.450 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.449+0000 7f75bd7fa640 1 -- 10.20.193.51:0/4036084301 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 11 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7f75c80ce0c0 con 0x7f7598078e00 2026-02-22T21:42:53.450 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 11 to deep-scrub 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 10 librados: watch_flush enter 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 10 librados: watch_flush exit 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 20 client.5332.objecter shutdown clearing up homeless session... 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 10 client.5332.objecter successfully canceled tick 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 msgr2=0x7f759807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 0x7f759807b2c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006dc0 tx=0x7f75b8009340 comp rx=0 tx=0).stop 2026-02-22T21:42:53.451 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 10 monclient: shutdown 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 msgr2=0x7f75c8155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f75c402f780 tx=0x7f75c4004b40 comp rx=0 tx=0).stop 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 shutdown_connections 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f7598078e00 0x7f759807b2c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f75c815c710 0x7f75c8155ec0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f75c8150600 0x7f75c814ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 --2- 10.20.193.51:0/4036084301 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f75c8150220 0x7f75c8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 >> 10.20.193.51:0/4036084301 conn(0x7f75c80cd7b0 msgr2=0x7f75c814cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 shutdown_connections 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 -- 10.20.193.51:0/4036084301 wait complete. 2026-02-22T21:42:53.452 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.450+0000 7f75cf20b640 1 librados: shutdown 2026-02-22T21:42:53.460 INFO:tasks.ceph:Scrubbing osd.15 2026-02-22T21:42:53.460 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 10 monclient: get_monmap_and_config 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 10 monclient: monmap: 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:53.529909+0000 2026-02-22T21:42:53.531 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:53.529909+0000 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 1 Processor -- start 2026-02-22T21:42:53.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 1 -- start start 2026-02-22T21:42:53.533 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.529+0000 7ff0f8274640 10 monclient: init 2026-02-22T21:42:53.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient: _add_conns ranks=[5,3,0,2,1,4] 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff0f0152910 0x7ff0f00c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-c con 0x7ff0f0152910 addr v2:10.20.193.125:3300/0 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7ff0f0152cf0 0x7ff0f0070810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-a con 0x7ff0f0152cf0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0f0159320 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f00c6390 con 0x7ff0f0159320 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f00737a0 con 0x7ff0f0152cf0 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f01597d0 con 0x7ff0f0152910 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f8274640 10 monclient(hunting): authenticate will time out at 1449.110317s 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff0f0152910 0x7ff0f00c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7ff0f0152910 0x7ff0f00c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:37062/0 (socket says 10.20.193.51:37062) 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/2684834475 learned_addr learned my addr 10.20.193.51:0/2684834475 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 10 monclient(hunting): get_auth_request con 0x7ff0f0152910 auth_method 0 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f5fe9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.530+0000 7ff0f57e8640 1 --2- 10.20.193.51:0/2684834475 >> v2:10.20.193.51:3300/0 conn(0x7ff0f0152cf0 0x7ff0f0070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.537 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): get_auth_request con 0x7ff0f0152cf0 auth_method 0 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.531+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_done global_id 5327 payload 274 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/2684834475 >> v2:10.20.193.51:3300/0 conn(0x7ff0f0152cf0 msgr2=0x7ff0f0070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 1 --2- 10.20.193.51:0/2684834475 >> v2:10.20.193.51:3300/0 conn(0x7ff0f0152cf0 0x7ff0f0070810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/2684834475 >> v1:10.20.193.51:6789/0 conn(0x7ff0f0159320 legacy=0x7ff0f00c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 1 monclient: found mon.noname-c 2026-02-22T21:42:53.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:42:53.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.532+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/2684834475 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f0073a60 con 0x7ff0f0152910 2026-02-22T21:42:53.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.533+0000 7ff0f5fe9640 1 --2- 10.20.193.51:0/2684834475 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0152910 0x7ff0f00c7c50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0069a40 tx=0x7ff0e002eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c90e397c67450e server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 1 -- 10.20.193.51:0/2684834475 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e003c070 con 0x7ff0f0152910 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: dump: 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.545 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 1 -- 10.20.193.51:0/2684834475 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e002fbe0 con 0x7ff0f0152910 2026-02-22T21:42:53.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 5 monclient: authenticate success, global_id 5327 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 1 -- 10.20.193.51:0/2684834475 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0037440 con 0x7ff0f0152910 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 4 set_mon_vals no callback set 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f4fe7640 10 monclient: dump: 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.547 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 10 monclient: shutdown 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 -- 10.20.193.51:0/2684834475 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0152910 msgr2=0x7ff0f00c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 --2- 10.20.193.51:0/2684834475 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0152910 0x7ff0f00c7c50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0069a40 tx=0x7ff0e002eed0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 -- 10.20.193.51:0/2684834475 shutdown_connections 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 --2- 10.20.193.51:0/2684834475 >> v2:10.20.193.51:3300/0 conn(0x7ff0f0152cf0 0x7ff0f0070810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 --2- 10.20.193.51:0/2684834475 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0152910 0x7ff0f00c7c50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 -- 10.20.193.51:0/2684834475 >> 10.20.193.51:0/2684834475 conn(0x7ff0f00c9d50 msgr2=0x7ff0f00ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 -- 10.20.193.51:0/2684834475 shutdown_connections 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 1 -- 10.20.193.51:0/2684834475 wait complete. 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.535+0000 7ff0f8274640 10 monclient: monmap: 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:53.549 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 librados: starting msgr at 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 librados: starting objecter 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 Processor -- start 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 -- start start 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 librados: setting wanted keys 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 librados: calling monclient init 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient: init 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient: _add_conns ranks=[0,1,2] 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 0x7ff0f0154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-a con 0x7ff0f0152910 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff0f0152cf0 0x7ff0f01f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-b con 0x7ff0f0152cf0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0159320 0x7ff0f0157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): picked mon.noname-c con 0x7ff0f0159320 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0074300 con 0x7ff0f0152910 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0074020 con 0x7ff0f0152cf0 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f00745e0 con 0x7ff0f0159320 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f8274640 10 monclient(hunting): authenticate will time out at 1449.116606s 2026-02-22T21:42:53.551 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 0x7ff0f0154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 0x7ff0f0154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35724/0 (socket says 10.20.193.51:35724) 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/50750964 learned_addr learned my addr 10.20.193.51:0/50750964 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 10 monclient(hunting): get_auth_request con 0x7ff0f0152910 auth_method 0 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f5fe9640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.536+0000 7ff0f67ea640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0159320 0x7ff0f0157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): get_auth_request con 0x7ff0f0159320 auth_method 0 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.552 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f67ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff0f0152cf0 0x7ff0f01f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): get_auth_request con 0x7ff0f0152cf0 auth_method 0 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.553 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.537+0000 7ff0f57e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 10 monclient(hunting): handle_auth_done global_id 5325 payload 824 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/50750964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0159320 msgr2=0x7ff0f0157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7ff0f0159320 0x7ff0f0157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/50750964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff0f0152cf0 msgr2=0x7ff0f01f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7ff0f0152cf0 0x7ff0f01f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.538+0000 7ff0f5fe9640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f01f6c70 con 0x7ff0f0152910 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.539+0000 7ff0f5fe9640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 0x7ff0f0154d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009a70 tx=0x7ff0e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.554 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.539+0000 7ff0de7fc640 10 client.?.objecter ms_handle_connect 0x7ff0f0152910 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.539+0000 7ff0de7fc640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e003c040 con 0x7ff0f0152910 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: dump: 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.555 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 5 monclient: authenticate success, global_id 5325 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e0042630 con 0x7ff0f0152910 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0042a20 con 0x7ff0f0152910 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 monclient: _renew_subs 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0dffff640 4 set_mon_vals no callback set 2026-02-22T21:42:53.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f01f42c0 con 0x7ff0f0152910 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0de7fc640 10 monclient: dump: 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.557 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 client.5325.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 monclient: _renew_subs 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f01f71f0 con 0x7ff0f0152910 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 1 librados: init done 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 client.5325.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 15 client.5325.objecter _session_command_op_assign -1 1 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 20 client.5325.objecter _get_session osd=-1 returning homeless 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 client.5325.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.541+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0f0074b00 con 0x7ff0f0152910 2026-02-22T21:42:53.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.553+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7ff0e0042bc0 con 0x7ff0f0152910 2026-02-22T21:42:53.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.554+0000 7ff0de7fc640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff0c0078e50 0x7ff0c007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.554+0000 7ff0f57e8640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff0c0078e50 0x7ff0c007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.554+0000 7ff0f57e8640 10 monclient: get_auth_request con 0x7ff0c0078e50 auth_method 0 2026-02-22T21:42:53.561 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.554+0000 7ff0f57e8640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff0c0078e50 0x7ff0c007b310 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0f006ecb0 tx=0x7ff0e4009290 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.554+0000 7ff0de7fc640 10 client.5325.objecter ms_handle_connect 0x7ff0c0078e50 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7ff0e002faa0 con 0x7ff0f0152910 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 10 client.5325.objecter ms_dispatch2 0x7ff0f0000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 3 client.5325.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 3 client.5325.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 10 client.5325.objecter checking command 1 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7ff0c0081330 0x7ff0c0083750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter _get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 15 client.5325.objecter _session_command_op_remove -1 1 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter _get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.562 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 15 client.5325.objecter _session_command_op_assign 15 1 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 4 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 10 client.5325.objecter _send_command 1 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0e000af30 con 0x7ff0c0081330 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 20 client.5325.objecter dump_active .. 0 homeless 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff0e0083990 con 0x7ff0f0152910 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.555+0000 7ff0de7fc640 10 monclient: handle_get_version_reply finishing 1 version 31 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.556+0000 7ff0f67ea640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7ff0c0081330 0x7ff0c0083750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.556+0000 7ff0f67ea640 10 monclient: get_auth_request con 0x7ff0c0081330 auth_method 0 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.558+0000 7ff0f67ea640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7ff0c0081330 0x7ff0c0083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.15 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.563 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.558+0000 7ff0de7fc640 10 client.5325.objecter ms_handle_connect 0x7ff0c0081330 2026-02-22T21:42:53.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.567+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== osd.15 v2:10.20.193.140:6800/2657947218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27504 (crc 0 0 0) 0x7ff0e000af30 con 0x7ff0c0081330 2026-02-22T21:42:53.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.567+0000 7ff0de7fc640 10 client.5325.objecter ms_dispatch2 0x7ff0f0000b90 command_reply(tid 1: 0 ) 2026-02-22T21:42:53.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.567+0000 7ff0de7fc640 10 client.5325.objecter _finish_command 1 = system:0 2026-02-22T21:42:53.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.567+0000 7ff0de7fc640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 4 2026-02-22T21:42:53.569 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.567+0000 7ff0de7fc640 15 client.5325.objecter _session_command_op_remove 15 1 2026-02-22T21:42:53.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 10 client.5325.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2026-02-22T21:42:53.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 15 client.5325.objecter _session_command_op_assign -1 2 2026-02-22T21:42:53.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 20 client.5325.objecter _get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.575 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 20 client.5325.objecter _get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 15 client.5325.objecter _session_command_op_remove -1 2 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 20 client.5325.objecter get_session s=0x7ff0c0080c60 osd=15 3 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 15 client.5325.objecter _session_command_op_assign 15 2 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 4 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 10 client.5325.objecter _send_command 2 2026-02-22T21:42:53.576 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.573+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 --> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff0f00c8850 con 0x7ff0c0081330 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0de7fc640 1 -- 10.20.193.51:0/50750964 <== osd.15 v2:10.20.193.140:6800/2657947218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff0f00c8850 con 0x7ff0c0081330 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0de7fc640 10 client.5325.objecter ms_dispatch2 0x7ff0f0000b90 command_reply(tid 2: 0 ) 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0de7fc640 10 client.5325.objecter _finish_command 2 = system:0 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0de7fc640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 4 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0de7fc640 15 client.5325.objecter _session_command_op_remove 15 2 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stdout:{ 2026-02-22T21:42:53.582 INFO:teuthology.orchestra.run.trial051.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T21:42:53.583 INFO:teuthology.orchestra.run.trial051.stdout:} 2026-02-22T21:42:53.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 10 librados: watch_flush enter 2026-02-22T21:42:53.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 10 librados: watch_flush exit 2026-02-22T21:42:53.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 10 client.5325.objecter close_session for osd.15 2026-02-22T21:42:53.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7ff0c0081330 msgr2=0x7ff0c0083750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.584 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.140:6800/2657947218,v1:10.20.193.140:6801/2657947218] conn(0x7ff0c0081330 0x7ff0c0083750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 20 client.5325.objecter put_session s=0x7ff0c0080c60 osd=15 1 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 20 client.5325.objecter shutdown clearing up homeless session... 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 10 client.5325.objecter successfully canceled tick 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff0c0078e50 msgr2=0x7ff0c007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7ff0c0078e50 0x7ff0c007b310 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0f006ecb0 tx=0x7ff0e4009290 comp rx=0 tx=0).stop 2026-02-22T21:42:53.585 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 10 monclient: shutdown 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 msgr2=0x7ff0f0154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 --2- 10.20.193.51:0/50750964 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7ff0f0152910 0x7ff0f0154d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009a70 tx=0x7ff0e00047c0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f67ea640 1 -- 10.20.193.51:0/50750964 reap_dead start 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 shutdown_connections 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 >> 10.20.193.51:0/50750964 conn(0x7ff0f00c9d50 msgr2=0x7ff0f006fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.586 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 shutdown_connections 2026-02-22T21:42:53.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 -- 10.20.193.51:0/50750964 wait complete. 2026-02-22T21:42:53.587 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.580+0000 7ff0f8274640 1 librados: shutdown 2026-02-22T21:42:53.591 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 10 monclient: get_monmap_and_config 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 10 monclient: monmap: 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:53.659812+0000 2026-02-22T21:42:53.661 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:53.659812+0000 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 1 Processor -- start 2026-02-22T21:42:53.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 1 -- start start 2026-02-22T21:42:53.663 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.659+0000 7f16fdb53640 10 monclient: init 2026-02-22T21:42:53.665 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient: _add_conns ranks=[3,0,1,2,5,4] 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f16f8152ab0 0x7f16f80c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-a con 0x7f16f8152ab0 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16f806d9d0 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f16f80c95a0 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f806f1d0 con 0x7f16f806d9d0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f806de50 con 0x7f16f80c95a0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f16f806ed40 con 0x7f16f8152ab0 2026-02-22T21:42:53.666 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16fdb53640 10 monclient(hunting): authenticate will time out at 1449.240227s 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f77fe640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f16f8152ab0 0x7f16f80c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f7fff640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f16f80c95a0 0x7f16f8158a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:39096/0 (socket says 10.20.193.51:39096) 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f7fff640 1 -- 10.20.193.51:0/1714834927 learned_addr learned my addr 10.20.193.51:0/1714834927 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16f8152ab0 auth_method 0 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.660+0000 7f16f77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.667 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 1 -- 10.20.193.51:0/1714834927 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538409746 0 0) 0x7f16f806de50 con 0x7f16f80c95a0 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 10 monclient(hunting): _negotiate 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 10 monclient(hunting): my global_id is 5338 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.661+0000 7f16f67fc640 1 -- 10.20.193.51:0/1714834927 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16dc003620 con 0x7f16f80c95a0 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 10 monclient(hunting): handle_auth_done global_id 5328 payload 274 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 1 -- 10.20.193.51:0/1714834927 >> v1:10.20.193.68:6789/0 conn(0x7f16f80c95a0 legacy=0x7f16f8158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 1 -- 10.20.193.51:0/1714834927 >> v1:10.20.193.51:6789/0 conn(0x7f16f806d9d0 legacy=0x7f16f80c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.668 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.663+0000 7f16f77fe640 1 -- 10.20.193.51:0/1714834927 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f806e130 con 0x7f16f8152ab0 2026-02-22T21:42:53.669 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.664+0000 7f16f77fe640 1 --2- 10.20.193.51:0/1714834927 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f8152ab0 0x7f16f80c5ab0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f16e40098d0 tx=0x7f16e402ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7515f68f95867453 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 1 -- 10.20.193.51:0/1714834927 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e403c070 con 0x7f16f8152ab0 2026-02-22T21:42:53.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.675 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: dump: 2026-02-22T21:42:53.675 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.676 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 1 -- 10.20.193.51:0/1714834927 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e402fbf0 con 0x7f16f8152ab0 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 1 -- 10.20.193.51:0/1714834927 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4037440 con 0x7f16f8152ab0 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 5 monclient: authenticate success, global_id 5328 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16f67fc640 10 monclient: dump: 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.677 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 4 set_mon_vals no callback set 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 10 monclient: shutdown 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.678 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 -- 10.20.193.51:0/1714834927 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f8152ab0 msgr2=0x7f16f80c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 --2- 10.20.193.51:0/1714834927 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f8152ab0 0x7f16f80c5ab0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f16e40098d0 tx=0x7f16e402ef20 comp rx=0 tx=0).stop 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 -- 10.20.193.51:0/1714834927 shutdown_connections 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 --2- 10.20.193.51:0/1714834927 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f8152ab0 0x7f16f80c5ab0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 -- 10.20.193.51:0/1714834927 >> 10.20.193.51:0/1714834927 conn(0x7f16f80cd7b0 msgr2=0x7f16f80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 -- 10.20.193.51:0/1714834927 shutdown_connections 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.666+0000 7f16fdb53640 1 -- 10.20.193.51:0/1714834927 wait complete. 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient: monmap: 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:53.679 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 librados: starting msgr at 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 librados: starting objecter 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 Processor -- start 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 -- start start 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 librados: setting wanted keys 2026-02-22T21:42:53.680 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 librados: calling monclient init 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient: init 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient: _add_conns ranks=[1,0,2] 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16f806d9d0 0x7f16f815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-b con 0x7f16f806d9d0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-a con 0x7f16f80c95a0 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16f8152ab0 0x7f16f81f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): picked mon.noname-c con 0x7f16f8152ab0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f16f806e9d0 con 0x7f16f80c95a0 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.681 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f16f806e6f0 con 0x7f16f806d9d0 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8152e90 con 0x7f16f8152ab0 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16fdb53640 10 monclient(hunting): authenticate will time out at 1449.247727s 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.667+0000 7f16f6ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35732/0 (socket says 10.20.193.51:35732) 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 1 -- 10.20.193.51:0/462724198 learned_addr learned my addr 10.20.193.51:0/462724198 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16f80c95a0 auth_method 0 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.682 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16f806d9d0 0x7f16f815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16f8152ab0 0x7f16f81f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16f806d9d0 auth_method 0 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16f8152ab0 auth_method 0 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.683 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.668+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 10 monclient(hunting): handle_auth_done global_id 5334 payload 824 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 -- 10.20.193.51:0/462724198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16f8152ab0 msgr2=0x7f16f81f7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16f8152ab0 0x7f16f81f7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 -- 10.20.193.51:0/462724198 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16f806d9d0 msgr2=0x7f16f815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.684 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16f806d9d0 0x7f16f815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.669+0000 7f16f6ffd640 1 -- 10.20.193.51:0/462724198 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f81f6c40 con 0x7f16f80c95a0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.670+0000 7f16f6ffd640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c760 tx=0x7f16e800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.670+0000 7f16d3fff640 10 client.?.objecter ms_handle_connect 0x7f16f80c95a0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.670+0000 7f16d3fff640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e80052d0 con 0x7f16f80c95a0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: dump: 2026-02-22T21:42:53.685 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.686 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e8005490 con 0x7f16f80c95a0 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e800a880 con 0x7f16f80c95a0 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16d3fff640 10 monclient: dump: 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.687 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16f5ffb640 4 set_mon_vals no callback set 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16fdb53640 5 monclient: authenticate success, global_id 5334 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16fdb53640 10 monclient: _renew_subs 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16fdb53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.671+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f81f3990 con 0x7f16f80c95a0 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 client.5334.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 monclient: _renew_subs 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.688 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f81f3f50 con 0x7f16f80c95a0 2026-02-22T21:42:53.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 1 librados: init done 2026-02-22T21:42:53.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.689 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.672+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f80c5ab0 con 0x7f16f80c95a0 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.683+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f16e800aa20 con 0x7f16f80c95a0 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.684+0000 7f16d3fff640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 0x7f16c807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.684+0000 7f16f77fe640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 0x7f16c807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.684+0000 7f16f77fe640 10 monclient: get_auth_request con 0x7f16c8078e00 auth_method 0 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.684+0000 7f16f77fe640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 0x7f16c807b2c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16e4039000 tx=0x7f16e4033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.690 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.684+0000 7f16d3fff640 10 client.5334.objecter ms_handle_connect 0x7f16c8078e00 2026-02-22T21:42:53.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.686+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f16e800ad40 con 0x7f16f80c95a0 2026-02-22T21:42:53.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.686+0000 7f16d3fff640 10 client.5334.objecter ms_dispatch2 0x7f16f8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:53.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.686+0000 7f16d3fff640 3 client.5334.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:53.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.686+0000 7f16d3fff640 3 client.5334.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:53.691 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.686+0000 7f16d3fff640 20 client.5334.objecter dump_active .. 0 homeless 2026-02-22T21:42:53.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.788+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f16e8062010 con 0x7f16f80c95a0 2026-02-22T21:42:53.789 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.788+0000 7f16d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.790 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.788+0000 7f16d3fff640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.869+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "15", "target": ["mon-mgr", ""]}) -- 0x7f16f81f3740 con 0x7f16c8078e00 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.869+0000 7f16d3fff640 1 -- 10.20.193.51:0/462724198 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 15 to deep-scrub) ==== 42+0+0 (secure 0 0 0) 0x7f16f81f3740 con 0x7f16c8078e00 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:instructed osd(s) 15 to deep-scrub 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 10 librados: watch_flush enter 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 10 librados: watch_flush exit 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 20 client.5334.objecter shutdown clearing up homeless session... 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 10 client.5334.objecter successfully canceled tick 2026-02-22T21:42:53.873 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 msgr2=0x7f16c807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 0x7f16c807b2c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16e4039000 tx=0x7f16e4033000 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 10 monclient: shutdown 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 msgr2=0x7f16f81485f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c760 tx=0x7f16e800cc30 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 shutdown_connections 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f16c8078e00 0x7f16c807b2c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f16f8152ab0 0x7f16f81f7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f16f80c95a0 0x7f16f81485f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 --2- 10.20.193.51:0/462724198 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f16f806d9d0 0x7f16f815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 >> 10.20.193.51:0/462724198 conn(0x7f16f80cd7b0 msgr2=0x7f16f80c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.874 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 shutdown_connections 2026-02-22T21:42:53.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 -- 10.20.193.51:0/462724198 wait complete. 2026-02-22T21:42:53.875 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.870+0000 7f16fdb53640 1 librados: shutdown 2026-02-22T21:42:53.880 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:42:53.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 10 monclient: get_monmap_and_config 2026-02-22T21:42:53.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.949 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 10 monclient: monmap: 2026-02-22T21:42:53.949 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:42:53.948084+0000 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:42:53.948084+0000 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:42:53.950 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 1 Processor -- start 2026-02-22T21:42:53.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 1 -- start start 2026-02-22T21:42:53.951 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.947+0000 7f9bd9656640 10 monclient: init 2026-02-22T21:42:53.953 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient: _add_conns ranks=[3,1,0,4,5,2] 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7f9bd4150220 0x7f9bd40c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-a con 0x7f9bd4150220 addr v2:10.20.193.51:3300/0 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7f9bd4150600 addr v1:10.20.193.68:6789/0 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bd415c710 addr v1:10.20.193.51:6789/0 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd415a920 con 0x7f9bd415c710 2026-02-22T21:42:53.954 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd415af60 con 0x7f9bd4150600 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd415b240 con 0x7f9bd4150220 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd9656640 10 monclient(hunting): authenticate will time out at 1449.528384s 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd27fc640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7f9bd4150600 0x7f9bd415c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:39110/0 (socket says 10.20.193.51:39110) 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.948+0000 7f9bd27fc640 1 -- 10.20.193.51:0/1311395308 learned_addr learned my addr 10.20.193.51:0/1311395308 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 1 --2- 10.20.193.51:0/1311395308 >> v2:10.20.193.51:3300/0 conn(0x7f9bd4150220 0x7f9bd40c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f9bd4150220 auth_method 0 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.955 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.949+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 1 -- 10.20.193.51:0/1311395308 <== mon.1 v1:10.20.193.68:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150812985 0 0) 0x7f9bd415af60 con 0x7f9bd4150600 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 10 monclient(hunting): _negotiate 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 10 monclient(hunting): my global_id is 5344 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.950+0000 7f9bd1ffb640 1 -- 10.20.193.51:0/1311395308 --> v1:10.20.193.68:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc0003620 con 0x7f9bd4150600 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_done global_id 5337 payload 274 2026-02-22T21:42:53.956 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 1 -- 10.20.193.51:0/1311395308 >> v1:10.20.193.68:6789/0 conn(0x7f9bd4150600 legacy=0x7f9bd415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 1 -- 10.20.193.51:0/1311395308 >> v1:10.20.193.51:6789/0 conn(0x7f9bd415c710 legacy=0x7f9bd406fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.957 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.951+0000 7f9bd2ffd640 1 -- 10.20.193.51:0/1311395308 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd415b520 con 0x7f9bd4150220 2026-02-22T21:42:53.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.952+0000 7f9bd2ffd640 1 --2- 10.20.193.51:0/1311395308 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd4150220 0x7f9bd40c73a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80098a0 tx=0x7f9bc802ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc328dfd654925b9 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 1 -- 10.20.193.51:0/1311395308 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc803c070 con 0x7f9bd4150220 2026-02-22T21:42:53.964 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: dump: 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.965 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd9656640 5 monclient: authenticate success, global_id 5337 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd9656640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 1 -- 10.20.193.51:0/1311395308 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc802fc10 con 0x7f9bd4150220 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.966 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 1 -- 10.20.193.51:0/1311395308 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8037440 con 0x7f9bd4150220 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd9656640 10 monclient: get_monmap_and_config success 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd9656640 4 set_mon_vals no callback set 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.954+0000 7f9bd1ffb640 10 monclient: dump: 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.967 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 10 monclient: shutdown 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- 10.20.193.51:0/1311395308 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd4150220 msgr2=0x7f9bd40c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 --2- 10.20.193.51:0/1311395308 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd4150220 0x7f9bd40c73a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80098a0 tx=0x7f9bc802ef00 comp rx=0 tx=0).stop 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- 10.20.193.51:0/1311395308 shutdown_connections 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 --2- 10.20.193.51:0/1311395308 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd4150220 0x7f9bd40c73a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- 10.20.193.51:0/1311395308 >> 10.20.193.51:0/1311395308 conn(0x7f9bd40cd7b0 msgr2=0x7f9bd40cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- 10.20.193.51:0/1311395308 shutdown_connections 2026-02-22T21:42:53.968 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- 10.20.193.51:0/1311395308 wait complete. 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 10 monclient: build_initial_monmap 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 10 monclient: monmap: 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.969 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 librados: starting msgr at 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 librados: starting objecter 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 Processor -- start 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 -- start start 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 librados: setting wanted keys 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 1 librados: calling monclient init 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.955+0000 7f9bd9656640 10 monclient: init 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient: _reopen_session rank -1 2026-02-22T21:42:53.970 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient: _add_conns ranks=[1,2,0] 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9bd4150220 0x7f9bd40732c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-b con 0x7f9bd4150220 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9bd4150600 0x7f9bd4072400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-c con 0x7f9bd4150600 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): picked mon.noname-a con 0x7f9bd415c710 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd40703b0 con 0x7f9bd415c710 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd415bae0 con 0x7f9bd4150220 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): start opening mon connection 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd4070530 con 0x7f9bd4150600 2026-02-22T21:42:53.971 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): _renew_subs 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd9656640 10 monclient(hunting): authenticate will time out at 1449.536161s 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd2ffd640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9bd4150220 0x7f9bd40732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:35750/0 (socket says 10.20.193.51:35750) 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 1 -- 10.20.193.51:0/3110347087 learned_addr learned my addr 10.20.193.51:0/3110347087 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): get_auth_request con 0x7f9bd415c710 auth_method 0 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd27fc640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9bd4150600 0x7f9bd4072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.972 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f9bd4150220 auth_method 0 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd2ffd640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd27fc640 10 monclient(hunting): get_auth_request con 0x7f9bd4150600 auth_method 0 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd27fc640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.956+0000 7f9bd27fc640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.973 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.957+0000 7f9bd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 10 monclient(hunting): handle_auth_done global_id 5343 payload 824 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 -- 10.20.193.51:0/3110347087 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9bd4150600 msgr2=0x7f9bd4072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9bd4150600 0x7f9bd4072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 -- 10.20.193.51:0/3110347087 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9bd4150220 msgr2=0x7f9bd40732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9bd4150220 0x7f9bd40732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 10 monclient: _finish_hunting 0 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 monclient: found mon.noname-a 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:42:53.974 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.958+0000 7f9bd37fe640 1 -- 10.20.193.51:0/3110347087 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd40be8b0 con 0x7f9bd415c710 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.959+0000 7f9bd37fe640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400cc50 tx=0x7f9bc400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.959+0000 7f9bb37fe640 10 client.?.objecter ms_handle_connect 0x7f9bd415c710 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.959+0000 7f9bb37fe640 10 client.?.objecter resend_mon_ops 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4003a70 con 0x7f9bd415c710 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: dump: 2026-02-22T21:42:53.975 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: _finish_auth 0 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: _check_auth_tickets 2026-02-22T21:42:53.976 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc4003c10 con 0x7f9bd415c710 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: handle_config config(0 keys) 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40050d0 con 0x7f9bd415c710 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 5 monclient: authenticate success, global_id 5343 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd17fa640 4 set_mon_vals no callback set 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bb37fe640 10 monclient: dump: 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:42:53.977 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 10 monclient: _renew_subs 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd40beb90 con 0x7f9bd415c710 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 10 client.5343.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 10 monclient: _renew_subs 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.978 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd40bf150 con 0x7f9bd415c710 2026-02-22T21:42:53.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.960+0000 7f9bd9656640 1 librados: init done 2026-02-22T21:42:53.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.961+0000 7f9bd9656640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.961+0000 7f9bd9656640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:53.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.961+0000 7f9bd9656640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:42:53.979 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.961+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd40c73a0 con 0x7f9bd415c710 2026-02-22T21:42:53.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.972+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7f9bc4005270 con 0x7f9bd415c710 2026-02-22T21:42:53.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.973+0000 7f9bb37fe640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 0x7f9ba007b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:42:53.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.973+0000 7f9bd2ffd640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 0x7f9ba007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:42:53.980 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.973+0000 7f9bd2ffd640 10 monclient: get_auth_request con 0x7f9ba0078e50 auth_method 0 2026-02-22T21:42:53.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.973+0000 7f9bd2ffd640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 0x7f9ba007b310 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8039000 tx=0x7f9bc8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:42:53.981 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:53.973+0000 7f9bb37fe640 10 client.5343.objecter ms_handle_connect 0x7f9ba0078e50 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7f9bc405e770 con 0x7f9bd415c710 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 10 client.5343.objecter ms_dispatch2 0x7f9bd4000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 3 client.5343.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 3 client.5343.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 20 client.5343.objecter dump_active .. 0 homeless 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7f9bc4062730 con 0x7f9bd415c710 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:42:54.076 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.075+0000 7f9bb37fe640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:42:54.157 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.155+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bd4074da0 con 0x7f9ba0078e50 2026-02-22T21:42:54.158 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.156+0000 7f9bb37fe640 1 -- 10.20.193.51:0/3110347087 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44364 (secure 0 0 0) 0x7f9bd4074da0 con 0x7f9ba0078e50 2026-02-22T21:42:54.159 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:42:54.160 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 10 librados: watch_flush enter 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 10 librados: watch_flush exit 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 20 client.5343.objecter shutdown clearing up homeless session... 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 10 client.5343.objecter successfully canceled tick 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 msgr2=0x7f9ba007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 0x7f9ba007b310 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8039000 tx=0x7f9bc8033000 comp rx=0 tx=0).stop 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 10 monclient: shutdown 2026-02-22T21:42:54.161 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 msgr2=0x7f9bd40cbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400cc50 tx=0x7f9bc400a430 comp rx=0 tx=0).stop 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 shutdown_connections 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7f9ba0078e50 0x7f9ba007b310 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7f9bd415c710 0x7f9bd40cbbb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7f9bd4150600 0x7f9bd4072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 --2- 10.20.193.51:0/3110347087 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7f9bd4150220 0x7f9bd40732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 >> 10.20.193.51:0/3110347087 conn(0x7f9bd40cd7b0 msgr2=0x7f9bd414da40 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 shutdown_connections 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 -- 10.20.193.51:0/3110347087 wait complete. 2026-02-22T21:42:54.162 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:42:54.157+0000 7f9bd9656640 1 librados: shutdown 2026-02-22T21:42:54.167 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2026-02-22T21:42:53.766692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437512,"kb_used_data":6104,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509511928,"statfs":{"total":1546188226560,"available":1545740214272,"internally_reserved":0,"allocated":6250496,"data_stored":3091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":127,"apply_latency_ms":127,"commit_latency_ns":127000000,"apply_latency_ns":127000000},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:44.312096+0000","last_change":"2026-02-22T21:37:36.846754+0000","last_active":"2026-02-22T21:39:44.312096+0000","last_peered":"2026-02-22T21:39:44.312096+0000","last_clean":"2026-02-22T21:39:44.312096+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:39:44.312096+0000","last_undegraded":"2026-02-22T21:39:44.312096+0000","last_fullsized":"2026-02-22T21:39:44.312096+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T06:07:58.604986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:42.253559+0000","last_change":"2026-02-22T21:37:36.795065+0000","last_active":"2026-02-22T21:39:42.253559+0000","last_peered":"2026-02-22T21:39:42.253559+0000","last_clean":"2026-02-22T21:39:42.253559+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:39:42.253559+0000","last_undegraded":"2026-02-22T21:39:42.253559+0000","last_fullsized":"2026-02-22T21:39:42.253559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:24:29.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:48.610582+0000","last_change":"2026-02-22T21:42:48.615227+0000","last_active":"2026-02-22T21:42:48.610582+0000","last_peered":"2026-02-22T21:42:48.610582+0000","last_clean":"2026-02-22T21:42:48.610582+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:42:48.610582+0000","last_undegraded":"2026-02-22T21:42:48.610582+0000","last_fullsized":"2026-02-22T21:42:48.610582+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:48.607095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:48.607095+0000","last_clean_scrub_stamp":"2026-02-22T21:42:48.607095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:21:59.837274+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:37.344398+0000","last_change":"2026-02-22T21:37:37.112984+0000","last_active":"2026-02-22T21:39:37.344398+0000","last_peered":"2026-02-22T21:39:37.344398+0000","last_clean":"2026-02-22T21:39:37.344398+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:39:37.344398+0000","last_undegraded":"2026-02-22T21:39:37.344398+0000","last_fullsized":"2026-02-22T21:39:37.344398+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:11:23.619429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"30'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:39.310121+0000","last_change":"2026-02-22T21:37:36.843543+0000","last_active":"2026-02-22T21:39:39.310121+0000","last_peered":"2026-02-22T21:39:39.310121+0000","last_clean":"2026-02-22T21:39:39.310121+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:39:39.310121+0000","last_undegraded":"2026-02-22T21:39:39.310121+0000","last_fullsized":"2026-02-22T21:39:39.310121+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:02:11.646735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:53.232915+0000","last_change":"2026-02-22T21:42:53.237637+0000","last_active":"2026-02-22T21:42:53.232915+0000","last_peered":"2026-02-22T21:42:53.232915+0000","last_clean":"2026-02-22T21:42:53.232915+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:42:53.232915+0000","last_undegraded":"2026-02-22T21:42:53.232915+0000","last_fullsized":"2026-02-22T21:42:53.232915+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:53.229598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:53.229598+0000","last_clean_scrub_stamp":"2026-02-22T21:42:53.229598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:10:12.479247+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:51.073274+0000","last_change":"2026-02-22T21:42:51.078248+0000","last_active":"2026-02-22T21:42:51.073274+0000","last_peered":"2026-02-22T21:42:51.073274+0000","last_clean":"2026-02-22T21:42:51.073274+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:42:51.073274+0000","last_undegraded":"2026-02-22T21:42:51.073274+0000","last_fullsized":"2026-02-22T21:42:51.073274+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:51.070061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:51.070061+0000","last_clean_scrub_stamp":"2026-02-22T21:42:51.070061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:53.191504+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"27'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:41:43.487427+0000","last_change":"2026-02-22T21:37:36.825527+0000","last_active":"2026-02-22T21:41:43.487427+0000","last_peered":"2026-02-22T21:41:43.487427+0000","last_clean":"2026-02-22T21:41:43.487427+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:41:43.487427+0000","last_undegraded":"2026-02-22T21:41:43.487427+0000","last_fullsized":"2026-02-22T21:41:43.487427+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:31.695978+0000","last_clean_scrub_stamp":"2026-02-22T21:37:31.695978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"30'32","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:39:28.703892+0000","last_change":"2026-02-22T21:37:30.261158+0000","last_active":"2026-02-22T21:39:28.703892+0000","last_peered":"2026-02-22T21:39:28.703892+0000","last_clean":"2026-02-22T21:39:28.703892+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:39:28.703892+0000","last_undegraded":"2026-02-22T21:39:28.703892+0000","last_fullsized":"2026-02-22T21:39:28.703892+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:37:28.122395+0000","last_clean_scrub_stamp":"2026-02-22T21:37:28.122395+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:56:50.203759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313285,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-22T21:37:31.695978+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=31, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-22T21:37:31.695978+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=31, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-22T21:37:31.695978+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=31, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-22T21:37:31.695978+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=31, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-22T21:37:31.695978+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=31, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T21:37:28.122395+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=37, tm_sec=28, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=21, tm_min=42, tm_sec=46, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T21:42:54.169 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T21:43:01.754 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:32.907 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.779 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:32.933 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.783 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:32.934 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.784 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:32.936 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.785 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:32.900 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.788 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:32.904 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.792 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:32.944 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.797 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:32.951 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:01.797 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:32.913 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:14.170 DEBUG:teuthology.orchestra.run.trial051:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T21:43:14.239 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.238+0000 7fcc78f33640 10 monclient: get_monmap_and_config 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.238+0000 7fcc78f33640 10 monclient: build_initial_monmap 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.238+0000 7fcc78f33640 10 monclient: monmap: 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:43:14.239087+0000 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:43:14.239087+0000 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:43:14.240 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:43:14.241 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:43:14.241 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:43:14.241 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.243 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.238+0000 7fcc78f33640 1 Processor -- start 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.238+0000 7fcc78f33640 1 -- start start 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient: init 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient: _reopen_session rank -1 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient: _add_conns ranks=[5,0,3,2,1,4] 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcc741526a0 0x7fcc740733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-c con 0x7fcc741526a0 addr v2:10.20.193.125:3300/0 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc74152a80 addr v1:10.20.193.51:6789/0 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fcc7415d3b0 0x7fcc7415d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-a con 0x7fcc7415d3b0 addr v2:10.20.193.51:3300/0 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 1 -- --> v1:10.20.193.51:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc74073d60 con 0x7fcc74152a80 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fcc740cca00 con 0x7fcc7415d3b0 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.244 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fcc74074c50 con 0x7fcc741526a0 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): _renew_subs 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.239+0000 7fcc78f33640 10 monclient(hunting): authenticate will time out at 1469.819570s 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcc741526a0 0x7fcc740733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fcc741526a0 0x7fcc740733d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.125:3300/0 says I am v2:10.20.193.51:45850/0 (socket says 10.20.193.51:45850) 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 1 -- 10.20.193.51:0/912200186 learned_addr learned my addr 10.20.193.51:0/912200186 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): get_auth_request con 0x7fcc741526a0 auth_method 0 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72d76640 1 --2- 10.20.193.51:0/912200186 >> v2:10.20.193.51:3300/0 conn(0x7fcc7415d3b0 0x7fcc7415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72d76640 10 monclient(hunting): get_auth_request con 0x7fcc7415d3b0 auth_method 0 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.245 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.240+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 10 monclient(hunting): handle_auth_done global_id 5339 payload 274 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 1 -- 10.20.193.51:0/912200186 >> v2:10.20.193.51:3300/0 conn(0x7fcc7415d3b0 msgr2=0x7fcc7415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 1 --2- 10.20.193.51:0/912200186 >> v2:10.20.193.51:3300/0 conn(0x7fcc7415d3b0 0x7fcc7415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 1 -- 10.20.193.51:0/912200186 >> v1:10.20.193.51:6789/0 conn(0x7fcc74152a80 legacy=0x7fcc740cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 10 monclient: _finish_hunting 0 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 1 monclient: found mon.noname-c 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 10 monclient: _send_mon_message to mon.noname-c at v2:10.20.193.125:3300/0 2026-02-22T21:43:14.246 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.241+0000 7fcc72575640 1 -- 10.20.193.51:0/912200186 --> v2:10.20.193.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc74074040 con 0x7fcc741526a0 2026-02-22T21:43:14.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.243+0000 7fcc72575640 1 --2- 10.20.193.51:0/912200186 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc740733d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcc7406cf40 tx=0x7fcc6802ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bca3dfbc6354a6 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 1 -- 10.20.193.51:0/912200186 <== mon.2 v2:10.20.193.125:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6803c070 con 0x7fcc741526a0 2026-02-22T21:43:14.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.250 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 10 monclient: dump: 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 10 monclient: _finish_auth 0 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 10 monclient: _check_auth_tickets 2026-02-22T21:43:14.251 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc71573640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc78f33640 5 monclient: authenticate success, global_id 5339 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.244+0000 7fcc78f33640 20 monclient: get_monmap_and_config waiting for monmap|config 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 1 -- 10.20.193.51:0/912200186 <== mon.2 v2:10.20.193.125:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc6802fc70 con 0x7fcc741526a0 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 10 monclient: handle_config config(0 keys) 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 1 -- 10.20.193.51:0/912200186 <== mon.2 v2:10.20.193.125:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68037660 con 0x7fcc741526a0 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 10 monclient: got monmap 1 from mon.c (according to old e1) 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc71573640 10 monclient: dump: 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.252 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 10 monclient: get_monmap_and_config success 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 4 set_mon_vals no callback set 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 10 monclient: shutdown 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 -- 10.20.193.51:0/912200186 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 msgr2=0x7fcc740733d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 --2- 10.20.193.51:0/912200186 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc740733d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcc7406cf40 tx=0x7fcc6802ef60 comp rx=0 tx=0).stop 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 -- 10.20.193.51:0/912200186 shutdown_connections 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 --2- 10.20.193.51:0/912200186 >> v2:10.20.193.51:3300/0 conn(0x7fcc7415d3b0 0x7fcc7415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 --2- 10.20.193.51:0/912200186 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc740733d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.253 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 -- 10.20.193.51:0/912200186 >> 10.20.193.51:0/912200186 conn(0x7fcc740ce350 msgr2=0x7fcc740ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 -- 10.20.193.51:0/912200186 shutdown_connections 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.245+0000 7fcc78f33640 1 -- 10.20.193.51:0/912200186 wait complete. 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 10 monclient: build_initial_monmap 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 10 monclient: monmap: 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 librados: starting msgr at 2026-02-22T21:43:14.254 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 librados: starting objecter 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 Processor -- start 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 -- start start 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 librados: setting wanted keys 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 1 librados: calling monclient init 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 10 monclient: init 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.246+0000 7fcc78f33640 10 monclient: _reopen_session rank -1 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc74158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-c con 0x7fcc741526a0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-a con 0x7fcc74152a80 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc7415d3b0 0x7fcc741577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): picked mon.noname-b con 0x7fcc7415d3b0 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.255 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fcc740748e0 con 0x7fcc74152a80 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fcc74074600 con 0x7fcc7415d3b0 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fcc740cdcf0 con 0x7fcc741526a0 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): _renew_subs 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc78f33640 10 monclient(hunting): authenticate will time out at 1469.827042s 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:57544/0 (socket says 10.20.193.51:57544) 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 1 -- 10.20.193.51:0/3588171646 learned_addr learned my addr 10.20.193.51:0/3588171646 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc74158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): get_auth_request con 0x7fcc74152a80 auth_method 0 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.256 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): get_auth_request con 0x7fcc741526a0 auth_method 0 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc71d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72d76640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc7415d3b0 0x7fcc741577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72d76640 10 monclient(hunting): get_auth_request con 0x7fcc7415d3b0 auth_method 0 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72d76640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.247+0000 7fcc72d76640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.257 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc72d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 10 monclient(hunting): handle_auth_done global_id 5352 payload 824 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 -- 10.20.193.51:0/3588171646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 msgr2=0x7fcc74158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc74158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 -- 10.20.193.51:0/3588171646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc7415d3b0 msgr2=0x7fcc741577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc7415d3b0 0x7fcc741577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 10 monclient: _finish_hunting 0 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 monclient: found mon.noname-a 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.258 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.248+0000 7fcc71d74640 1 -- 10.20.193.51:0/3588171646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc741567b0 con 0x7fcc74152a80 2026-02-22T21:43:14.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.249+0000 7fcc71d74640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c5b0 tx=0x7fcc5c00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.249+0000 7fcc5affd640 10 client.?.objecter ms_handle_connect 0x7fcc74152a80 2026-02-22T21:43:14.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.249+0000 7fcc5affd640 10 client.?.objecter resend_mon_ops 2026-02-22T21:43:14.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0052d0 con 0x7fcc74152a80 2026-02-22T21:43:14.260 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: dump: 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.261 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: _finish_auth 0 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: _check_auth_tickets 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc5c005470 con 0x7fcc74152a80 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: handle_config config(0 keys) 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0038f0 con 0x7fcc74152a80 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc70d72640 4 set_mon_vals no callback set 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc5affd640 10 monclient: dump: 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.262 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 5 monclient: authenticate success, global_id 5352 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 10 monclient: _renew_subs 2026-02-22T21:43:14.263 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7414b410 con 0x7fcc74152a80 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 10 client.5352.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 10 monclient: _renew_subs 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7414b9d0 con 0x7fcc74152a80 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.251+0000 7fcc78f33640 1 librados: init done 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.252+0000 7fcc78f33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.252+0000 7fcc78f33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.252+0000 7fcc78f33640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.264 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.252+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc74160600 con 0x7fcc74152a80 2026-02-22T21:43:14.265 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fcc5c003a90 con 0x7fcc74152a80 2026-02-22T21:43:14.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc5affd640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 0x7fcc4007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc72575640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 0x7fcc4007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc72575640 10 monclient: get_auth_request con 0x7fcc40078e00 auth_method 0 2026-02-22T21:43:14.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc5affd640 10 client.5352.objecter ms_handle_connect 0x7fcc40078e00 2026-02-22T21:43:14.266 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.264+0000 7fcc72575640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 0x7fcc4007b2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc7415f410 tx=0x7fcc68039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.266+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fcc5c05e020 con 0x7fcc74152a80 2026-02-22T21:43:14.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.266+0000 7fcc5affd640 10 client.5352.objecter ms_dispatch2 0x7fcc74000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:43:14.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.266+0000 7fcc5affd640 3 client.5352.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:43:14.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.266+0000 7fcc5affd640 3 client.5352.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:43:14.267 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.266+0000 7fcc5affd640 20 client.5352.objecter dump_active .. 0 homeless 2026-02-22T21:43:14.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.366+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fcc74160600 con 0x7fcc74152a80 2026-02-22T21:43:14.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.366+0000 7fcc5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.368 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.366+0000 7fcc5affd640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:43:14.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.452+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 --> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc7415ec50 con 0x7fcc40078e00 2026-02-22T21:43:14.453 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.452+0000 7fcc5affd640 1 -- 10.20.193.51:0/3588171646 <== mgr.4103 v2:10.20.193.51:6800/3302837800 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+44316 (secure 0 0 0) 0x7fcc7415ec50 con 0x7fcc40078e00 2026-02-22T21:43:14.454 INFO:teuthology.orchestra.run.trial051.stdout: 2026-02-22T21:43:14.454 INFO:teuthology.orchestra.run.trial051.stderr:dumped all 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 10 librados: watch_flush enter 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 10 librados: watch_flush exit 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 20 client.5352.objecter shutdown clearing up homeless session... 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 10 client.5352.objecter successfully canceled tick 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 msgr2=0x7fcc4007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 0x7fcc4007b2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc7415f410 tx=0x7fcc68039040 comp rx=0 tx=0).stop 2026-02-22T21:43:14.455 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 10 monclient: shutdown 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 msgr2=0x7fcc74157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c5b0 tx=0x7fcc5c00ca80 comp rx=0 tx=0).stop 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 shutdown_connections 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fcc40078e00 0x7fcc4007b2c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fcc7415d3b0 0x7fcc741577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fcc74152a80 0x7fcc74157270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 --2- 10.20.193.51:0/3588171646 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fcc741526a0 0x7fcc74158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 >> 10.20.193.51:0/3588171646 conn(0x7fcc740ce350 msgr2=0x7fcc7415f440 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 shutdown_connections 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 -- 10.20.193.51:0/3588171646 wait complete. 2026-02-22T21:43:14.456 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.454+0000 7fcc78f33640 1 librados: shutdown 2026-02-22T21:43:14.464 INFO:teuthology.orchestra.run.trial051.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-22T21:43:13.769690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":59,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":437488,"kb_used_data":6104,"kb_used_omap":24,"kb_used_meta":431079,"kb_avail":1509511952,"statfs":{"total":1546188226560,"available":1545740238848,"internally_reserved":0,"allocated":6250496,"data_stored":3091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25502,"internal_metadata":441424994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"," osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001782"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:51.028303+0000","last_change":"2026-02-22T21:42:51.032935+0000","last_active":"2026-02-22T21:42:51.028303+0000","last_peered":"2026-02-22T21:42:51.028303+0000","last_clean":"2026-02-22T21:42:51.028303+0000","last_became_active":"2026-02-22T21:37:33.160697+0000","last_became_peered":"2026-02-22T21:37:33.160697+0000","last_unstale":"2026-02-22T21:42:51.028303+0000","last_undegraded":"2026-02-22T21:42:51.028303+0000","last_fullsized":"2026-02-22T21:42:51.028303+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:51.024977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:51.024977+0000","last_clean_scrub_stamp":"2026-02-22T21:42:51.024977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:02:26.395972+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0.027421108999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:54.847557+0000","last_change":"2026-02-22T21:42:54.850260+0000","last_active":"2026-02-22T21:42:54.847557+0000","last_peered":"2026-02-22T21:42:54.847557+0000","last_clean":"2026-02-22T21:42:54.847557+0000","last_became_active":"2026-02-22T21:37:33.013679+0000","last_became_peered":"2026-02-22T21:37:33.013679+0000","last_unstale":"2026-02-22T21:42:54.847557+0000","last_undegraded":"2026-02-22T21:42:54.847557+0000","last_fullsized":"2026-02-22T21:42:54.847557+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:54.844329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:54.844329+0000","last_clean_scrub_stamp":"2026-02-22T21:42:54.844329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:44:52.577726+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.033519238,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:48.610582+0000","last_change":"2026-02-22T21:42:48.615227+0000","last_active":"2026-02-22T21:42:48.610582+0000","last_peered":"2026-02-22T21:42:48.610582+0000","last_clean":"2026-02-22T21:42:48.610582+0000","last_became_active":"2026-02-22T21:37:33.277273+0000","last_became_peered":"2026-02-22T21:37:33.277273+0000","last_unstale":"2026-02-22T21:42:48.610582+0000","last_undegraded":"2026-02-22T21:42:48.610582+0000","last_fullsized":"2026-02-22T21:42:48.610582+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:48.607095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:48.607095+0000","last_clean_scrub_stamp":"2026-02-22T21:42:48.607095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:21:59.837274+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0.037437501999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:54.425216+0000","last_change":"2026-02-22T21:42:54.430379+0000","last_active":"2026-02-22T21:42:54.425216+0000","last_peered":"2026-02-22T21:42:54.425216+0000","last_clean":"2026-02-22T21:42:54.425216+0000","last_became_active":"2026-02-22T21:37:32.978936+0000","last_became_peered":"2026-02-22T21:37:32.978936+0000","last_unstale":"2026-02-22T21:42:54.425216+0000","last_undegraded":"2026-02-22T21:42:54.425216+0000","last_fullsized":"2026-02-22T21:42:54.425216+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:54.420078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:54.420078+0000","last_clean_scrub_stamp":"2026-02-22T21:42:54.420078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T09:35:59.318608+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0.042250226000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"31'3","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:51.942819+0000","last_change":"2026-02-22T21:42:51.943782+0000","last_active":"2026-02-22T21:42:51.942819+0000","last_peered":"2026-02-22T21:42:51.942819+0000","last_clean":"2026-02-22T21:42:51.942819+0000","last_became_active":"2026-02-22T21:37:33.251405+0000","last_became_peered":"2026-02-22T21:37:33.251405+0000","last_unstale":"2026-02-22T21:42:51.942819+0000","last_undegraded":"2026-02-22T21:42:51.942819+0000","last_fullsized":"2026-02-22T21:42:51.942819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2026-02-22T21:42:51.941405+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2026-02-22T21:42:51.941405+0000","last_clean_scrub_stamp":"2026-02-22T21:42:51.941405+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:46:29.880605+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0.055776341,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:53.232915+0000","last_change":"2026-02-22T21:42:53.237637+0000","last_active":"2026-02-22T21:42:53.232915+0000","last_peered":"2026-02-22T21:42:53.232915+0000","last_clean":"2026-02-22T21:42:53.232915+0000","last_became_active":"2026-02-22T21:37:32.980366+0000","last_became_peered":"2026-02-22T21:37:32.980366+0000","last_unstale":"2026-02-22T21:42:53.232915+0000","last_undegraded":"2026-02-22T21:42:53.232915+0000","last_fullsized":"2026-02-22T21:42:53.232915+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:53.229598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:53.229598+0000","last_clean_scrub_stamp":"2026-02-22T21:42:53.229598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-23T22:10:12.479247+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0.046036372999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,15],"acting":[14,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:51.073274+0000","last_change":"2026-02-22T21:42:51.078248+0000","last_active":"2026-02-22T21:42:51.073274+0000","last_peered":"2026-02-22T21:42:51.073274+0000","last_clean":"2026-02-22T21:42:51.073274+0000","last_became_active":"2026-02-22T21:37:33.012561+0000","last_became_peered":"2026-02-22T21:37:33.012561+0000","last_unstale":"2026-02-22T21:42:51.073274+0000","last_undegraded":"2026-02-22T21:42:51.073274+0000","last_fullsized":"2026-02-22T21:42:51.073274+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T21:42:51.070061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T21:42:51.070061+0000","last_clean_scrub_stamp":"2026-02-22T21:42:51.070061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T05:41:53.191504+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0.037981304,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"31'2","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:54.240273+0000","last_change":"2026-02-22T21:42:54.241141+0000","last_active":"2026-02-22T21:42:54.240273+0000","last_peered":"2026-02-22T21:42:54.240273+0000","last_clean":"2026-02-22T21:42:54.240273+0000","last_became_active":"2026-02-22T21:37:33.521546+0000","last_became_peered":"2026-02-22T21:37:33.521546+0000","last_unstale":"2026-02-22T21:42:54.240273+0000","last_undegraded":"2026-02-22T21:42:54.240273+0000","last_fullsized":"2026-02-22T21:42:54.240273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2026-02-22T21:42:54.238290+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2026-02-22T21:42:54.238290+0000","last_clean_scrub_stamp":"2026-02-22T21:42:54.238290+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T02:57:29.858406+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0.033196278000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"31'34","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-22T21:42:52.305836+0000","last_change":"2026-02-22T21:42:52.311122+0000","last_active":"2026-02-22T21:42:52.305836+0000","last_peered":"2026-02-22T21:42:52.305836+0000","last_clean":"2026-02-22T21:42:52.305836+0000","last_became_active":"2026-02-22T21:37:30.095556+0000","last_became_peered":"2026-02-22T21:37:30.095556+0000","last_unstale":"2026-02-22T21:42:52.305836+0000","last_undegraded":"2026-02-22T21:42:52.305836+0000","last_fullsized":"2026-02-22T21:42:52.305836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'34","last_scrub_stamp":"2026-02-22T21:42:52.302737+0000","last_deep_scrub":"31'34","last_deep_scrub_stamp":"2026-02-22T21:42:52.302737+0000","last_clean_scrub_stamp":"2026-02-22T21:42:52.302737+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:53:23.304152+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":37,"num_write":57,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":24,"seq":103079215176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":24,"seq":103079215176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[1,3,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 8 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":1,"up_from":24,"seq":103079215175,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3,4,5,6,7,8,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 1 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":7,"up_from":22,"seq":94489280584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27589067},"hb_peers":[0,1,2,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 7 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27589061},"hb_peers":[0,1,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313289,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,1,3,4,5,6,7,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 2 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 0 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":13,"up_from":23,"seq":98784247881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":344064,"data_stored":143492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 13 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":3,"up_from":23,"seq":98784247880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2,4,6,7,9,10,11,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 3 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":4,"up_from":23,"seq":98784247880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[3,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 4 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":5,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27589065},"hb_peers":[4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 5 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":6,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":741376,"data_stored":541313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27589063},"hb_peers":[0,5,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 6 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":9,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[0,2,4,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 9 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":10,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27589057},"hb_peers":[2,4,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[" osd: 10 BLUESTORE_SLOW_OP_ALERT:observed slow operation indications in BlueStore"]},{"osd":14,"up_from":23,"seq":98784247880,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":339968,"data_stored":143473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3,4,5,6,7,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T21:43:14.465 DEBUG:teuthology.orchestra.run.trial051:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.530+0000 7fe5df4da640 10 monclient: get_monmap_and_config 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.530+0000 7fe5df4da640 10 monclient: build_initial_monmap 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.530+0000 7fe5df4da640 10 monclient: monmap: 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:43:14.531607+0000 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:43:14.531607+0000 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:0: v1:10.20.193.51:6789/0 mon.noname-a-legacy 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:1: v1:10.20.193.68:6789/0 mon.noname-b-legacy 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:2: v1:10.20.193.125:6789/0 mon.noname-c-legacy 2026-02-22T21:43:14.532 INFO:teuthology.orchestra.run.trial051.stderr:3: v2:10.20.193.51:3300/0 mon.noname-a 2026-02-22T21:43:14.533 INFO:teuthology.orchestra.run.trial051.stderr:4: v2:10.20.193.68:3300/0 mon.noname-b 2026-02-22T21:43:14.533 INFO:teuthology.orchestra.run.trial051.stderr:5: v2:10.20.193.125:3300/0 mon.noname-c 2026-02-22T21:43:14.533 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.531+0000 7fe5df4da640 1 Processor -- start 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.531+0000 7fe5df4da640 1 -- start start 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.531+0000 7fe5df4da640 10 monclient: init 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient: _reopen_session rank -1 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient: _add_conns ranks=[3,1,5,0,4,2] 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe5d80c9100 0x7fe5d80c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d80c9100 addr v2:10.20.193.51:3300/0 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-b-legacy con 0x7fe5d806d9b0 addr v1:10.20.193.68:6789/0 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe5d80c6230 0x7fe5d8152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d80c6230 addr v2:10.20.193.125:3300/0 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 1 -- --> v1:10.20.193.68:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5d806e1c0 con 0x7fe5d806d9b0 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 1 -- --> v2:10.20.193.51:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d806e4a0 con 0x7fe5d80c9100 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.534 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 1 -- --> v2:10.20.193.125:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d806e780 con 0x7fe5d80c6230 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): _renew_subs 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5df4da640 10 monclient(hunting): authenticate will time out at 1470.112190s 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dd24f640 1 --2- >> v2:10.20.193.51:3300/0 conn(0x7fe5d80c9100 0x7fe5d80c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dda50640 1 --2- >> v2:10.20.193.125:3300/0 conn(0x7fe5d80c6230 0x7fe5d8152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dca4e640 1 --1- >> v1:10.20.193.68:6789/0 conn(0x7fe5d806d9b0 0x7fe5d80c5b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:10.20.193.68:6789/0 says I am v1:10.20.193.51:46108/0 (socket says 10.20.193.51:46108) 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dca4e640 1 -- 10.20.193.51:0/3079146287 learned_addr learned my addr 10.20.193.51:0/3079146287 (peer_addr_for_me v1:10.20.193.51:0/0) 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dd24f640 10 monclient(hunting): get_auth_request con 0x7fe5d80c9100 auth_method 0 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dd24f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dd24f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dd24f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dda50640 10 monclient(hunting): get_auth_request con 0x7fe5d80c6230 auth_method 0 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dda50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dda50640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.532+0000 7fe5dda50640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.535 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.533+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_done global_id 5355 payload 274 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 1 -- 10.20.193.51:0/3079146287 >> v2:10.20.193.125:3300/0 conn(0x7fe5d80c6230 msgr2=0x7fe5d8152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 1 --2- 10.20.193.51:0/3079146287 >> v2:10.20.193.125:3300/0 conn(0x7fe5d80c6230 0x7fe5d8152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 1 -- 10.20.193.51:0/3079146287 >> v1:10.20.193.68:6789/0 conn(0x7fe5d806d9b0 legacy=0x7fe5d80c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 10 monclient: _finish_hunting 0 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 1 monclient: found mon.noname-a 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.534+0000 7fe5dd24f640 1 -- 10.20.193.51:0/3079146287 --> v2:10.20.193.51:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d806ea60 con 0x7fe5d80c9100 2026-02-22T21:43:14.536 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.535+0000 7fe5dd24f640 1 --2- 10.20.193.51:0/3079146287 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c9100 0x7fe5d80c94e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80c54b0 tx=0x7fe5c802eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb7a8d1087b8796 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 1 -- 10.20.193.51:0/3079146287 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c803c070 con 0x7fe5d80c9100 2026-02-22T21:43:14.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.538 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: dump: 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: _finish_auth 0 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: _check_auth_tickets 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 1 -- 10.20.193.51:0/3079146287 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c802fb80 con 0x7fe5d80c9100 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: handle_config config(0 keys) 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 1 -- 10.20.193.51:0/3079146287 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8037440 con 0x7fe5d80c9100 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 5 monclient: authenticate success, global_id 5355 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 10 monclient: get_monmap_and_config success 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 4 set_mon_vals no callback set 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5c7fff640 10 monclient: dump: 2026-02-22T21:43:14.539 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 10 monclient: shutdown 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 -- 10.20.193.51:0/3079146287 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c9100 msgr2=0x7fe5d80c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 --2- 10.20.193.51:0/3079146287 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c9100 0x7fe5d80c94e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80c54b0 tx=0x7fe5c802eeb0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 -- 10.20.193.51:0/3079146287 shutdown_connections 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 --2- 10.20.193.51:0/3079146287 >> v2:10.20.193.125:3300/0 conn(0x7fe5d80c6230 0x7fe5d8152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 --2- 10.20.193.51:0/3079146287 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c9100 0x7fe5d80c94e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 -- 10.20.193.51:0/3079146287 >> 10.20.193.51:0/3079146287 conn(0x7fe5d80cd7b0 msgr2=0x7fe5d80cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 -- 10.20.193.51:0/3079146287 shutdown_connections 2026-02-22T21:43:14.540 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 1 -- 10.20.193.51:0/3079146287 wait complete. 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 10 monclient: build_initial_monmap 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.537+0000 7fe5df4da640 10 monclient: monmap: 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:epoch 0 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:fsid 00000000-0000-0000-0000-000000000000 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 0.000000 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:created 0.000000 2026-02-22T21:43:14.542 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 0 (unknown) 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.noname-a 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.noname-b 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.noname-c 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 librados: starting msgr at 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 librados: starting objecter 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 Processor -- start 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 -- start start 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 librados: setting wanted keys 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 librados: calling monclient init 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient: init 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient: _reopen_session rank -1 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient: _add_conns ranks=[2,0,1] 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5d806d9b0 0x7fe5d81f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d806d9b0 addr [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d80c6230 addr [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 --2- >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5d80c9100 0x7fe5d80736c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d80c9100 addr [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 -- --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d806ddb0 con 0x7fe5d80c6230 2026-02-22T21:43:14.543 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 -- --> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d806f020 con 0x7fe5d80c9100 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): start opening mon connection 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 1 -- --> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d806df30 con 0x7fe5d806d9b0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): _renew_subs 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5df4da640 10 monclient(hunting): authenticate will time out at 1470.118644s 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.538+0000 7fe5dca4e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 1 --2- >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5d806d9b0 0x7fe5d81f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 1 --2- >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:10.20.193.51:3300/0 says I am v2:10.20.193.51:57574/0 (socket says 10.20.193.51:57574) 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 1 -- 10.20.193.51:0/1494220719 learned_addr learned my addr 10.20.193.51:0/1494220719 (peer_addr_for_me v2:10.20.193.51:0/0) 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): get_auth_request con 0x7fe5d80c6230 auth_method 0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5d80c9100 0x7fe5d80736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): get_auth_request con 0x7fe5d806d9b0 auth_method 0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dca4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): get_auth_request con 0x7fe5d80c9100 auth_method 0 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): _init_auth method 2 2026-02-22T21:43:14.544 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): _init_auth creating new auth 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dda50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more payload 9 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.539+0000 7fe5dd24f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 10 monclient(hunting): handle_auth_done global_id 5358 payload 824 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 -- 10.20.193.51:0/1494220719 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5d806d9b0 msgr2=0x7fe5d81f00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5d806d9b0 0x7fe5d81f00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 -- 10.20.193.51:0/1494220719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5d80c9100 msgr2=0x7fe5d80736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5d80c9100 0x7fe5d80736c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 10 monclient: _finish_hunting 0 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 monclient: found mon.noname-a 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 10 monclient: _send_mon_message to mon.noname-a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.545 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.540+0000 7fe5dca4e640 1 -- 10.20.193.51:0/1494220719 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d8158700 con 0x7fe5d80c6230 2026-02-22T21:43:14.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.541+0000 7fe5dca4e640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c760 tx=0x7fe5cc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.541+0000 7fe5c5ffb640 10 client.?.objecter ms_handle_connect 0x7fe5d80c6230 2026-02-22T21:43:14.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.541+0000 7fe5c5ffb640 10 client.?.objecter resend_mon_ops 2026-02-22T21:43:14.546 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc017070 con 0x7fe5d80c6230 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: dump: 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.547 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: _finish_auth 0 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: _check_auth_tickets 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5cc005b90 con 0x7fe5d80c6230 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: handle_config config(0 keys) 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc007070 con 0x7fe5d80c6230 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2026-02-22T21:43:14.548 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c77fe640 4 set_mon_vals no callback set 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5c5ffb640 10 monclient: dump: 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:epoch 1 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:fsid f64be66c-6ce4-4d13-a1b8-cf6040bfed5c 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:last_changed 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:created 2026-02-22T21:34:07.089457+0000 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:min_mon_release 19 (squid) 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:election_strategy: 1 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:0: [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] mon.a 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:1: [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] mon.b 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:2: [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] mon.c 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:43:14.549 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 5 monclient: authenticate success, global_id 5358 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 monclient: _renew_subs 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5d81568a0 con 0x7fe5d80c6230 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 client.5358.objecter _maybe_request_map subscribing (onetime) to next osd map 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 monclient: _renew_subs 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5d8156e60 con 0x7fe5d80c6230 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 1 librados: init done 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.543+0000 7fe5df4da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.544+0000 7fe5df4da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.544+0000 7fe5df4da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.550 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.544+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5d806a210 con 0x7fe5d80c6230 2026-02-22T21:43:14.556 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.555+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 4 ==== mgrmap(e 5) ==== 99901+0+0 (secure 0 0 0) 0x7fe5cc007210 con 0x7fe5d80c6230 2026-02-22T21:43:14.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.555+0000 7fe5c5ffb640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 0x7fe5a807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2026-02-22T21:43:14.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.555+0000 7fe5dd24f640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 0x7fe5a807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2026-02-22T21:43:14.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.555+0000 7fe5dd24f640 10 monclient: get_auth_request con 0x7fe5a8078e00 auth_method 0 2026-02-22T21:43:14.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.556+0000 7fe5dd24f640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 0x7fe5a807b2c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8039000 tx=0x7fe5c8033000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2026-02-22T21:43:14.557 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.556+0000 7fe5c5ffb640 10 client.5358.objecter ms_handle_connect 0x7fe5a8078e00 2026-02-22T21:43:14.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.557+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 8944+0+0 (secure 0 0 0) 0x7fe5cc05d4a0 con 0x7fe5d80c6230 2026-02-22T21:43:14.558 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.557+0000 7fe5c5ffb640 10 client.5358.objecter ms_dispatch2 0x7fe5d8000b90 osd_map(31..31 src has 1..31) 2026-02-22T21:43:14.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.557+0000 7fe5c5ffb640 3 client.5358.objecter handle_osd_map got epochs [31,31] > 0 2026-02-22T21:43:14.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.557+0000 7fe5c5ffb640 3 client.5358.objecter handle_osd_map decoding full epoch 31 2026-02-22T21:43:14.559 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.557+0000 7fe5c5ffb640 20 client.5358.objecter dump_active .. 0 homeless 2026-02-22T21:43:14.661 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.660+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195034 (secure 0 0 0) 0x7fe5d806a210 con 0x7fe5d80c6230 2026-02-22T21:43:14.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.660+0000 7fe5c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2026-02-22T21:43:14.662 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.660+0000 7fe5c5ffb640 10 monclient: _finish_command 1 = system:0 2026-02-22T21:43:14.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.743+0000 7fe5df4da640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2026-02-22T21:43:14.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.743+0000 7fe5df4da640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2026-02-22T21:43:14.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.743+0000 7fe5df4da640 10 monclient: _send_mon_message to mon.a at v2:10.20.193.51:3300/0 2026-02-22T21:43:14.745 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.743+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 --> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fe5d80c94e0 con 0x7fe5d80c6230 2026-02-22T21:43:14.807 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) ==== 123+0+0 (secure 0 0 0) 0x7fe5cc021020 con 0x7fe5d80c6230 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c5ffb640 10 monclient: _finish_command 2 = system:0 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c5ffb640 1 -- 10.20.193.51:0/1494220719 <== mon.0 v2:10.20.193.51:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe5cc061460 con 0x7fe5d80c6230 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c5ffb640 10 monclient: handle_config config(1 keys) 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c77fe640 4 set_mon_vals no callback set 2026-02-22T21:43:14.808 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.806+0000 7fe5c77fe640 10 set_mon_vals mon_health_to_clog = false 2026-02-22T21:43:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 10 librados: watch_flush enter 2026-02-22T21:43:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 10 librados: watch_flush exit 2026-02-22T21:43:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 20 client.5358.objecter shutdown clearing up homeless session... 2026-02-22T21:43:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 10 client.5358.objecter successfully canceled tick 2026-02-22T21:43:14.810 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 msgr2=0x7fe5a807b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 0x7fe5a807b2c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8039000 tx=0x7fe5c8033000 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 10 monclient: shutdown 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 20 monclient: shutdown discarding 0 pending message(s) 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 msgr2=0x7fe5d81581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c760 tx=0x7fe5cc00cc30 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 shutdown_connections 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:6800/3302837800,v1:10.20.193.51:6801/3302837800] conn(0x7fe5a8078e00 0x7fe5a807b2c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.68:3300/0,v1:10.20.193.68:6789/0] conn(0x7fe5d80c9100 0x7fe5d80736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.51:3300/0,v1:10.20.193.51:6789/0] conn(0x7fe5d80c6230 0x7fe5d81581c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 --2- 10.20.193.51:0/1494220719 >> [v2:10.20.193.125:3300/0,v1:10.20.193.125:6789/0] conn(0x7fe5d806d9b0 0x7fe5d81f00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2026-02-22T21:43:14.811 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 >> 10.20.193.51:0/1494220719 conn(0x7fe5d80cd7b0 msgr2=0x7fe5d81591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2026-02-22T21:43:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 shutdown_connections 2026-02-22T21:43:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 -- 10.20.193.51:0/1494220719 wait complete. 2026-02-22T21:43:14.812 INFO:teuthology.orchestra.run.trial051.stderr:2026-02-22T21:43:14.808+0000 7fe5df4da640 1 librados: shutdown 2026-02-22T21:43:14.818 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 1922, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 1484, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph_manager.py", line 3303, in wait_until_healthy assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: timeout expired in wait_until_healthy 2026-02-22T21:43:14.821 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T21:43:14.821 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T21:43:14.822 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T21:43:14.822 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:14.829 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:43:14.823+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 45061) UID: 0 2026-02-22T21:43:14.830 INFO:tasks.ceph.osd.0.trial051.stderr:2026-02-22T21:43:14.824+0000 beae640 -1 osd.0 31 *** Got signal Terminated *** 2026-02-22T21:43:20.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-22T21:43:20.923 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T21:43:20.924 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2026-02-22T21:43:20.924 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:20.930 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:43:20.925+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 45064) UID: 0 2026-02-22T21:43:20.930 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:43:20.925+0000 beae640 -1 osd.4 33 *** Got signal Terminated *** 2026-02-22T21:43:21.681 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:43:21.679+0000 9e98640 -1 monclient: get_auth_request but no auth handler is set up 2026-02-22T21:43:21.687 INFO:tasks.ceph.osd.4.trial051.stderr:2026-02-22T21:43:21.686+0000 a699640 -1 monclient: get_auth_request but no auth handler is set up 2026-02-22T21:43:21.812 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:52.965 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.814 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:52.968 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.825 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:07:52.979 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.841 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:52.992 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.841 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:52.957 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.842 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:52.994 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.842 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:52.958 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.851 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:07:53.003 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:21.852 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:07:52.968 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:27.025 INFO:tasks.ceph.osd.4:Stopped 2026-02-22T21:43:27.025 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2026-02-22T21:43:27.025 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:27.033 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:43:27.027+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.8.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 8 (PID: 45065) UID: 0 2026-02-22T21:43:27.034 INFO:tasks.ceph.osd.8.trial051.stderr:2026-02-22T21:43:27.028+0000 beae640 -1 osd.8 34 *** Got signal Terminated *** 2026-02-22T21:43:27.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2026-02-22T21:43:27.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-22T21:43:33.127 INFO:tasks.ceph.osd.8:Stopped 2026-02-22T21:43:33.127 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2026-02-22T21:43:33.128 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:33.135 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:43:33.129+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.12.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 12 (PID: 45068) UID: 0 2026-02-22T21:43:33.136 INFO:tasks.ceph.osd.12.trial051.stderr:2026-02-22T21:43:33.130+0000 beae640 -1 osd.12 36 *** Got signal Terminated *** 2026-02-22T21:43:34.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2026-02-22T21:43:34.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2026-02-22T21:43:39.230 INFO:tasks.ceph.osd.12:Stopped 2026-02-22T21:43:39.230 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T21:43:39.230 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:39.237 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:43:39.227+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 44628) UID: 0 2026-02-22T21:43:39.237 INFO:tasks.ceph.osd.1.trial068.stderr:2026-02-22T21:43:39.228+0000 beae640 -1 osd.1 37 *** Got signal Terminated *** 2026-02-22T21:43:40.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~0s 2026-02-22T21:43:40.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 has been restored 2026-02-22T21:43:41.830 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:12.984 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.833 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:12.987 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.844 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:12.997 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.859 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:13.010 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.859 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:12.975 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.861 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:13.012 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.861 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:12.977 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.871 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:13.022 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:41.871 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:12.987 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:43:45.332 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T21:43:45.333 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2026-02-22T21:43:45.333 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:45.340 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:43:45.330+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 44627) UID: 0 2026-02-22T21:43:45.340 INFO:tasks.ceph.osd.5.trial068.stderr:2026-02-22T21:43:45.331+0000 beae640 -1 osd.5 38 *** Got signal Terminated *** 2026-02-22T21:43:47.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 has been restored 2026-02-22T21:43:51.435 INFO:tasks.ceph.osd.5:Stopped 2026-02-22T21:43:51.435 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2026-02-22T21:43:51.436 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:51.443 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:43:51.433+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.9.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 9 (PID: 44631) UID: 0 2026-02-22T21:43:51.444 INFO:tasks.ceph.osd.9.trial068.stderr:2026-02-22T21:43:51.434+0000 beae640 -1 osd.9 40 *** Got signal Terminated *** 2026-02-22T21:43:57.537 INFO:tasks.ceph.osd.9:Stopped 2026-02-22T21:43:57.538 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2026-02-22T21:43:57.538 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:43:57.546 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:43:57.536+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.13.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 13 (PID: 44660) UID: 0 2026-02-22T21:43:57.546 INFO:tasks.ceph.osd.13.trial068.stderr:2026-02-22T21:43:57.536+0000 beae640 -1 osd.13 41 *** Got signal Terminated *** 2026-02-22T21:44:01.869 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:33.023 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.871 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:33.022 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.871 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:33.025 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.873 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:33.025 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.882 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:33.036 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.883 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:33.034 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.896 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:33.012 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.898 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:33.014 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:01.908 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:33.024 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:03.640 INFO:tasks.ceph.osd.13:Stopped 2026-02-22T21:44:03.640 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T21:44:03.640 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:03.647 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:44:03.641+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 44659) UID: 0 2026-02-22T21:44:03.647 INFO:tasks.ceph.osd.2.trial125.stderr:2026-02-22T21:44:03.641+0000 beae640 -1 osd.2 43 *** Got signal Terminated *** 2026-02-22T21:44:09.741 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T21:44:09.742 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2026-02-22T21:44:09.742 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:09.749 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:44:09.742+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 44661) UID: 0 2026-02-22T21:44:09.749 INFO:tasks.ceph.osd.6.trial125.stderr:2026-02-22T21:44:09.743+0000 beae640 -1 osd.6 44 *** Got signal Terminated *** 2026-02-22T21:44:19.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2026-02-22T21:44:21.875 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:53.029 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.901 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:53.054 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.901 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:53.052 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.902 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:53.054 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.903 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:53.018 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.904 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:53.020 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.911 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:08:53.065 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.911 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:08:53.063 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.913 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:08:53.029 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:21.944 INFO:tasks.ceph.osd.6:Stopped 2026-02-22T21:44:21.945 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2026-02-22T21:44:21.945 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:21.952 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:44:21.945+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.10.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 10 (PID: 44664) UID: 0 2026-02-22T21:44:21.952 INFO:tasks.ceph.osd.10.trial125.stderr:2026-02-22T21:44:21.946+0000 beae640 -1 osd.10 45 *** Got signal Terminated *** 2026-02-22T21:44:25.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2026-02-22T21:44:28.047 INFO:tasks.ceph.osd.10:Stopped 2026-02-22T21:44:28.047 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2026-02-22T21:44:28.047 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:28.054 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:44:28.048+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.14.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 14 (PID: 44665) UID: 0 2026-02-22T21:44:28.054 INFO:tasks.ceph.osd.14.trial125.stderr:2026-02-22T21:44:28.049+0000 beae640 -1 osd.14 46 *** Got signal Terminated *** 2026-02-22T21:44:34.148 INFO:tasks.ceph.osd.14:Stopped 2026-02-22T21:44:34.149 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-22T21:44:34.149 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:34.167 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:44:34.159+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 45026) UID: 0 2026-02-22T21:44:34.168 INFO:tasks.ceph.osd.3.trial140.stderr:2026-02-22T21:44:34.160+0000 beae640 -1 osd.3 48 *** Got signal Terminated *** 2026-02-22T21:44:36.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2026-02-22T21:44:40.251 INFO:tasks.ceph.osd.3:Stopped 2026-02-22T21:44:40.251 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2026-02-22T21:44:40.252 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:40.258 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:44:40.250+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 45027) UID: 0 2026-02-22T21:44:40.260 INFO:tasks.ceph.osd.7.trial140.stderr:2026-02-22T21:44:40.251+0000 beae640 -1 osd.7 48 *** Got signal Terminated *** 2026-02-22T21:44:41.904 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:13.058 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.929 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:13.045 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.930 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:13.081 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.930 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:13.084 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.931 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:13.047 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.932 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:13.083 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.940 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:13.094 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.940 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:13.056 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:41.941 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:13.093 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:44:42.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2026-02-22T21:44:52.455 INFO:tasks.ceph.osd.7:Stopped 2026-02-22T21:44:52.455 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2026-02-22T21:44:52.455 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:52.467 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:44:52.457+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.11.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 11 (PID: 45029) UID: 0 2026-02-22T21:44:52.468 INFO:tasks.ceph.osd.11.trial140.stderr:2026-02-22T21:44:52.457+0000 beae640 -1 osd.11 48 *** Got signal Terminated *** 2026-02-22T21:44:54.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2026-02-22T21:44:58.557 INFO:tasks.ceph.osd.11:Stopped 2026-02-22T21:44:58.557 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2026-02-22T21:44:58.557 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:44:58.565 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:44:58.556+0000 beae640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/osd.15.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 15 (PID: 45031) UID: 0 2026-02-22T21:44:58.566 INFO:tasks.ceph.osd.15.trial140.stderr:2026-02-22T21:44:58.557+0000 beae640 -1 osd.15 48 *** Got signal Terminated *** 2026-02-22T21:44:59.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2026-02-22T21:45:01.933 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:33.087 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.958 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:33.109 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.958 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:33.112 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.958 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:33.074 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.960 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:33.111 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.960 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:33.076 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.967 INFO:tasks.ceph.mon.b.trial068.stderr:==00:00:09:33.119 44415== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.968 INFO:tasks.ceph.mon.a.trial051.stderr:==00:00:09:33.122 44639== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:01.968 INFO:tasks.ceph.mon.c.trial125.stderr:==00:00:09:33.084 44425== Warning: unimplemented fcntl command: 1036 2026-02-22T21:45:04.660 INFO:tasks.ceph.osd.15:Stopped 2026-02-22T21:45:04.660 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T21:45:04.660 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2026-02-22T21:45:04.661 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:45:04.661 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:45:04.660+0000 7f62a85f2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 44688) UID: 0 2026-02-22T21:45:04.661 INFO:tasks.ceph.mgr.y.trial051.stderr:2026-02-22T21:45:04.660+0000 7f62a85f2640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-22T21:45:04.693 INFO:tasks.ceph.mgr.y:Stopped 2026-02-22T21:45:04.693 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-22T21:45:04.693 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:45:04.725 INFO:tasks.ceph.mgr.x:Stopped 2026-02-22T21:45:04.725 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T21:45:04.725 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-22T21:45:04.725 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:45:04.741 INFO:tasks.ceph.mon.a.trial051.stderr:2026-02-22T21:45:04.735+0000 aca6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 44637) UID: 0 2026-02-22T21:45:04.741 INFO:tasks.ceph.mon.a.trial051.stderr:2026-02-22T21:45:04.735+0000 aca6640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-22T21:45:10.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2026-02-22T21:45:10.826 INFO:tasks.ceph.mon.a:Stopped 2026-02-22T21:45:10.826 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-22T21:45:10.826 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:45:10.834 INFO:tasks.ceph.mon.b.trial068.stderr:2026-02-22T21:45:10.823+0000 aca6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 44413) UID: 0 2026-02-22T21:45:10.835 INFO:tasks.ceph.mon.b.trial068.stderr:2026-02-22T21:45:10.824+0000 aca6640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2026-02-22T21:45:15.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2026-02-22T21:45:15.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2026-02-22T21:45:16.928 INFO:tasks.ceph.mon.b:Stopped 2026-02-22T21:45:16.928 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-22T21:45:16.928 INFO:teuthology.orchestra.run:waiting for 1200 2026-02-22T21:45:16.936 INFO:tasks.ceph.mon.c.trial125.stderr:2026-02-22T21:45:16.929+0000 aca6640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --gen-suppressions=all --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 44423) UID: 0 2026-02-22T21:45:16.937 INFO:tasks.ceph.mon.c.trial125.stderr:2026-02-22T21:45:16.930+0000 aca6640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2026-02-22T21:45:20.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2026-02-22T21:45:20.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2026-02-22T21:45:23.029 INFO:tasks.ceph.mon.c:Stopped 2026-02-22T21:45:23.030 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T21:45:23.030 DEBUG:teuthology.orchestra.run.trial051:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v 'OSD bench result' | head -n 1 2026-02-22T21:45:23.060 INFO:teuthology.orchestra.run.trial051.stdout:2026-02-22T21:37:28.298282+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2026-02-22T21:45:23.062 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2026-02-22T21:45:23.062 DEBUG:teuthology.orchestra.run.trial051:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v 'OSD bench result' | head -n 1 2026-02-22T21:45:23.128 DEBUG:teuthology.orchestra.run.trial051:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v 'OSD bench result' | head -n 1 2026-02-22T21:45:23.193 DEBUG:teuthology.orchestra.run.trial051:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v 'OSD bench result' | head -n 1 2026-02-22T21:45:23.259 INFO:teuthology.orchestra.run.trial051.stdout:2026-02-22T21:37:28.298282+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) 2026-02-22T21:45:23.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:45:23.260 DEBUG:teuthology.orchestra.run.trial051:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-22T21:45:23.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:45:23.329 DEBUG:teuthology.orchestra.run.trial051:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2026-02-22T21:45:23.403 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:45:23.403 DEBUG:teuthology.orchestra.run.trial051:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2026-02-22T21:45:23.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:45:23.476 DEBUG:teuthology.orchestra.run.trial051:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2026-02-22T21:45:23.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:45:23.544 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-22T21:45:23.583 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:45:23.584 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2026-02-22T21:45:23.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:45:23.657 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2026-02-22T21:45:23.727 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:45:23.727 DEBUG:teuthology.orchestra.run.trial068:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2026-02-22T21:45:23.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:45:23.802 DEBUG:teuthology.orchestra.run.trial125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-22T21:45:23.835 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:45:23.835 DEBUG:teuthology.orchestra.run.trial125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2026-02-22T21:45:23.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:45:23.903 DEBUG:teuthology.orchestra.run.trial125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2026-02-22T21:45:23.971 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:45:23.971 DEBUG:teuthology.orchestra.run.trial125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2026-02-22T21:45:24.038 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:45:24.038 DEBUG:teuthology.orchestra.run.trial140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-22T21:45:24.072 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:45:24.073 DEBUG:teuthology.orchestra.run.trial140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2026-02-22T21:45:24.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:45:24.140 DEBUG:teuthology.orchestra.run.trial140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2026-02-22T21:45:24.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:45:24.207 DEBUG:teuthology.orchestra.run.trial140:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2026-02-22T21:45:24.275 INFO:tasks.ceph:Archiving mon data... 2026-02-22T21:45:24.277 DEBUG:teuthology.misc:Transferring archived files from trial051:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/data/mon.a.tgz 2026-02-22T21:45:24.277 DEBUG:teuthology.orchestra.run.trial051:> mktemp 2026-02-22T21:45:24.296 INFO:teuthology.orchestra.run.trial051.stdout:/tmp/tmp.DclAfJ3WUW 2026-02-22T21:45:24.296 DEBUG:teuthology.orchestra.run.trial051:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DclAfJ3WUW 2026-02-22T21:45:24.407 DEBUG:teuthology.orchestra.run.trial051:> sudo chmod 0666 /tmp/tmp.DclAfJ3WUW 2026-02-22T21:45:24.488 DEBUG:teuthology.orchestra.remote:trial051:/tmp/tmp.DclAfJ3WUW is 412KB 2026-02-22T21:45:24.578 DEBUG:teuthology.orchestra.run.trial051:> rm -fr /tmp/tmp.DclAfJ3WUW 2026-02-22T21:45:24.595 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/data/mon.b.tgz 2026-02-22T21:45:24.595 DEBUG:teuthology.orchestra.run.trial068:> mktemp 2026-02-22T21:45:24.613 INFO:teuthology.orchestra.run.trial068.stdout:/tmp/tmp.jMTEeBTd94 2026-02-22T21:45:24.614 DEBUG:teuthology.orchestra.run.trial068:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jMTEeBTd94 2026-02-22T21:45:24.723 DEBUG:teuthology.orchestra.run.trial068:> sudo chmod 0666 /tmp/tmp.jMTEeBTd94 2026-02-22T21:45:24.803 DEBUG:teuthology.orchestra.remote:trial068:/tmp/tmp.jMTEeBTd94 is 412KB 2026-02-22T21:45:24.871 DEBUG:teuthology.orchestra.run.trial068:> rm -fr /tmp/tmp.jMTEeBTd94 2026-02-22T21:45:24.888 DEBUG:teuthology.misc:Transferring archived files from trial125:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/data/mon.c.tgz 2026-02-22T21:45:24.888 DEBUG:teuthology.orchestra.run.trial125:> mktemp 2026-02-22T21:45:24.906 INFO:teuthology.orchestra.run.trial125.stdout:/tmp/tmp.jqgymhtCvR 2026-02-22T21:45:24.907 DEBUG:teuthology.orchestra.run.trial125:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jqgymhtCvR 2026-02-22T21:45:25.017 DEBUG:teuthology.orchestra.run.trial125:> sudo chmod 0666 /tmp/tmp.jqgymhtCvR 2026-02-22T21:45:25.099 DEBUG:teuthology.orchestra.remote:trial125:/tmp/tmp.jqgymhtCvR is 412KB 2026-02-22T21:45:25.161 DEBUG:teuthology.orchestra.run.trial125:> rm -fr /tmp/tmp.jqgymhtCvR 2026-02-22T21:45:25.177 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T21:45:25.178 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T21:45:25.181 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T21:45:25.185 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T21:45:25.219 DEBUG:teuthology.orchestra.run.trial140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T21:45:25.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2026-02-22T21:45:25.484 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable malloc malloc _dl_new_object 2026-02-22T21:45:26.119 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T21:45:26.120 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T21:45:26.122 DEBUG:teuthology.misc:Transferring archived files from trial051:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial051/crash 2026-02-22T21:45:26.123 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T21:45:26.150 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial068/crash 2026-02-22T21:45:26.152 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T21:45:26.178 DEBUG:teuthology.misc:Transferring archived files from trial125:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial125/crash 2026-02-22T21:45:26.179 DEBUG:teuthology.orchestra.run.trial125:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T21:45:26.205 DEBUG:teuthology.misc:Transferring archived files from trial140:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial140/crash 2026-02-22T21:45:26.206 DEBUG:teuthology.orchestra.run.trial140:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T21:45:26.232 INFO:tasks.ceph:Compressing logs... 2026-02-22T21:45:26.232 DEBUG:teuthology.orchestra.run.trial051:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:26.236 DEBUG:teuthology.orchestra.run.trial068:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:26.238 DEBUG:teuthology.orchestra.run.trial125:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:26.249 DEBUG:teuthology.orchestra.run.trial140:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:26.269 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.13.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr: 68.2% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/valgrind/osd.13.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.9.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr: 68.2% -- replaced with /var/log/ceph/valgrind/osd.13.log.gz 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr: 67.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2026-02-22T21:45:26.270 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/valgrind/osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr: 68.1% -- replaced with /var/log/ceph/valgrind/osd.9.log.gz 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr: 94.1% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph.log: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T21:45:26.271 INFO:teuthology.orchestra.run.trial068.stderr:/var/log/ceph/ceph-osd.9.log: 89.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T21:45:26.280 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-22T21:45:26.280 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-22T21:45:26.280 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2026-02-22T21:45:26.280 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2026-02-22T21:45:26.280 INFO:teuthology.orchestra.run.trial125.stderr:gzip/var/log/ceph/valgrind/osd.2.log: -5 --verbose -- /var/log/ceph/valgrind/osd.14.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr: 68.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.10.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/valgrind/osd.6.log: 67.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/valgrind/osd.14.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr: 68.1% -- replaced with /var/log/ceph/valgrind/osd.14.log.gz 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/valgrind/osd.10.log: 68.2% -- replaced with /var/log/ceph/valgrind/osd.10.log.gz 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/valgrind/mon.c.log: /var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T21:45:26.281 INFO:teuthology.orchestra.run.trial125.stderr: 94.1% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2026-02-22T21:45:26.282 INFO:teuthology.orchestra.run.trial125.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.log: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T21:45:26.282 INFO:teuthology.orchestra.run.trial125.stderr: 89.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T21:45:26.303 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44719.log 2026-02-22T21:45:26.303 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-22T21:45:26.303 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44191.log 2026-02-22T21:45:26.303 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-osd.3.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.44719.log.gz --verbose 2026-02-22T21:45:26.303 INFO:teuthology.orchestra.run.trial140.stderr: -- /var/log/ceph/ceph-client.admin.44672.log 2026-02-22T21:45:26.312 INFO:teuthology.orchestra.run.trial068.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-22T21:45:26.313 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44191.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2026-02-22T21:45:26.313 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44672.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2026-02-22T21:45:26.313 INFO:teuthology.orchestra.run.trial140.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44191.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.11.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.15.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/valgrind/osd.7.log: /var/log/ceph/valgrind/osd.11.log: 68.3% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 68.2% -- replaced with /var/log/ceph/valgrind/osd.11.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/valgrind/osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 67.8% -- replaced with /var/log/ceph/valgrind/osd.15.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44390.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 68.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44531.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44343.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44390.log: /var/log/ceph/ceph-client.admin.44531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44907.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44343.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44531.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44390.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44484.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44343.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44437.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44907.log: /var/log/ceph/ceph-client.admin.44484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44437.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44484.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44578.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44907.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44625.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44437.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44578.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz --verbose 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr: -- /var/log/ceph/ceph-client.admin.44766.log 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44625.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44578.log.gz 2026-02-22T21:45:26.314 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44625.log.gz 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44813.log 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44240.log 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-mgr.x.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44813.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44813.log.gz 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr:/var/log/ceph/ceph-client.admin.44240.log: /var/log/ceph/ceph-osd.15.log: /var/log/ceph/ceph-client.admin.44860.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44240.log.gz 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2026-02-22T21:45:26.315 INFO:teuthology.orchestra.run.trial140.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-22T21:45:26.317 INFO:teuthology.orchestra.run.trial068.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48695.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55269.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55269.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48695.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51937.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49789.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55269.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54681.log: /var/log/ceph/ceph-client.admin.51937.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51937.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56592.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58356.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49789.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51133.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56592.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51843.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58356.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50344.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51843.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51133.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51843.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48649.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56837.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50344.log.gz 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51922.log 2026-02-22T21:45:26.319 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48649.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54828.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57768.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56837.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54828.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51922.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54828.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip/var/log/ceph/ceph-client.admin.57768.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49319.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57768.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49152.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49319.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54779.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49319.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57180.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51420.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49152.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57180.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54779.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51420.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57180.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54189.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51012.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51420.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50606.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54189.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57229.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54189.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58503.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57229.log: gzip -5 --verbose 87.4% -- /var/log/ceph/ceph-client.admin.49368.log -- replaced with /var/log/ceph/ceph-client.admin.50606.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58503.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57229.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49783.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49368.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58503.log.gz 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58797.log 2026-02-22T21:45:26.320 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49783.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.52282.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49368.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58797.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49783.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55808.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58797.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49813.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55808.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57866.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49813.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55808.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51538.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56445.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51538.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57866.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58894.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51538.log.gz/var/log/ceph/ceph-client.admin.56445.log: 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54483.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58894.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56445.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57621.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58894.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54287.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54483.log: /var/log/ceph/ceph-client.admin.57621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52798.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54483.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54287.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57621.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49817.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54287.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49817.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52798.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44690.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49817.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49777.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.44690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48525.log 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.44690.log.gz 2026-02-22T21:45:26.321 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49777.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.46332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53944.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48525.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58209.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48525.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49803.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53944.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49803.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58209.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.45100.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49803.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52602.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52183.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52602.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52183.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52602.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50583.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50605.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52183.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58111.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50583.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54632.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58111.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50583.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38341.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58111.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51011.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph.tmp-client.admin.38341.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38341.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51265.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54632.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51011.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51265.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51265.log.gz 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52321.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.12.log 2026-02-22T21:45:26.322 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.8.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 68.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/valgrind/osd.12.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 68.1% -- replaced with /var/log/ceph/valgrind/osd.12.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/valgrind/osd.8.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 68.2% -- replaced with /var/log/ceph/valgrind/osd.8.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46231.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57964.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 68.2% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 94.1% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.46231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46231.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50603.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51646.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57964.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50603.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49820.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.50603.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48787.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51646.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52959.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48787.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49820.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52504.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52959.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48787.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54877.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52504.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52959.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54582.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54877.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52504.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48879.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54582.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54877.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50564.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54582.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56543.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48879.log.gz 2026-02-22T21:45:26.323 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56543.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50564.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56543.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56200.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49791.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52455.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56200.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55563.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52455.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56200.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55906.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53401.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52455.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55906.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55563.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58650.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55906.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip/var/log/ceph/ceph-client.admin.53401.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56984.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53401.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56984.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58650.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50529.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56984.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55857.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50529.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54434.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 87.2%/var/log/ceph/ceph-client.admin.55857.log: -- replaced with /var/log/ceph/ceph-client.admin.50529.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55122.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57327.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54434.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57327.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55122.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53451.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57327.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53205.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53451.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2026-02-22T21:45:26.324 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53205.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53451.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-osd.12.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53205.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53550.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51418.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57523.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51418.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53550.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54532.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51418.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50601.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57523.log: /var/log/ceph/ceph-client.admin.54532.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57523.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50601.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54532.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58062.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56053.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58552.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56053.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58062.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52048.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56053.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52847.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58552.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 87.3%/var/log/ceph/ceph-client.admin.52847.log: -- replaced with /var/log/ceph/ceph-client.admin.52048.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51296.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 87.6%/var/log/ceph/ceph-client.admin.52651.log: -- replaced with /var/log/ceph/ceph-client.admin.52847.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55710.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51296.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53254.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55710.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51296.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53648.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53254.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55710.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57474.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53254.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49270.log 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53648.log.gz 2026-02-22T21:45:26.325 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51132.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57474.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49800.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56739.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 87.9%/var/log/ceph/ceph-client.admin.49800.log: -- replaced with /var/log/ceph/ceph-client.admin.49270.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53156.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56739.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49800.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54336.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53156.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56739.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51206.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53156.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54336.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58160.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54336.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51206.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58307.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52700.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58307.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58160.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54730.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 90.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52700.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58307.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57719.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52700.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54975.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54730.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49081.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52896.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57719.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49081.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54975.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58847.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49081.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55318.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52896.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55465.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53008.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58847.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55465.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55318.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53008.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55465.log.gz 2026-02-22T21:45:26.326 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51043.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53008.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.45169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58601.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55416.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58601.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55220.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58601.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49189.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48741.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55416.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49189.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55220.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56935.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.48741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55514.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49189.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56935.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48741.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56935.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58405.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.45123.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55514.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55955.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58405.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57082.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58405.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55955.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56151.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57082.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56102.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57033.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58013.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57033.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56102.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53845.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57033.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52102.log 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53845.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58013.log.gz 2026-02-22T21:45:26.327 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57670.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52102.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53845.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57572.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52102.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53795.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58699.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57670.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53795.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.57572.log.gz /var/log/ceph/ceph-client.admin.57376.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58699.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53795.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52184.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58699.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54385.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53303.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57376.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54385.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52184.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53993.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54385.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53303.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56004.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50596.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53993.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.6%/var/log/ceph/ceph-client.admin.56004.log: -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50596.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56004.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54238.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50596.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53895.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54140.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53895.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54238.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51539.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54140.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53895.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54091.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58748.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54140.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54091.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51539.log.gz 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45336.log 2026-02-22T21:45:26.328 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54091.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53599.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.45336.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58748.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49808.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53599.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45336.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54926.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53599.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55612.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52182.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49808.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55612.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54926.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57425.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52182.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55612.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53107.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57425.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52182.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50558.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53107.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57425.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51540.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50558.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53107.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52749.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50558.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51540.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51516.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52749.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56298.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56347.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53501.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56298.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57278.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56298.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55024.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57278.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53501.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49779.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55024.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57278.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58454.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49779.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55024.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49822.log 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58454.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49779.log.gz 2026-02-22T21:45:26.329 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56396.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49822.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58454.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52553.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56396.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49822.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53057.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53697.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56396.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53057.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52553.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56886.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53697.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53057.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56690.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56886.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53697.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56690.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56886.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55661.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56690.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53746.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50587.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.53746.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55661.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50599.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 89.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53746.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55171.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50587.log: /var/log/ceph/ceph-client.admin.50599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55367.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50587.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50599.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55171.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55367.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55171.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55073.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57817.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-osd.8.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55367.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.55073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55073.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51769.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56641.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57817.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49807.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.51769.log: /var/log/ceph/ceph-client.admin.49807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56249.log 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51769.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-22T21:45:26.330 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.50591.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49807.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54042.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56249.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57131.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.54042.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56249.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57131.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54042.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56494.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57131.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50598.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.56494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52406.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr: 88.8%/var/log/ceph/ceph-client.admin.50598.log: -- replaced with /var/log/ceph/ceph-client.admin.56494.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58258.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50598.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.52406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57915.log 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.58258.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52406.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr:/var/log/ceph/ceph-client.admin.57915.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58258.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial051.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57915.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial125.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial068.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2026-02-22T21:45:26.331 INFO:teuthology.orchestra.run.trial068.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2026-02-22T21:45:26.334 INFO:teuthology.orchestra.run.trial051.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2026-02-22T21:45:26.336 INFO:teuthology.orchestra.run.trial125.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2026-02-22T21:45:26.337 INFO:teuthology.orchestra.run.trial051.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2026-02-22T21:45:26.337 INFO:teuthology.orchestra.run.trial125.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2026-02-22T21:45:26.339 INFO:teuthology.orchestra.run.trial125.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2026-02-22T21:45:26.352 INFO:teuthology.orchestra.run.trial140.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-22T21:45:26.358 INFO:teuthology.orchestra.run.trial051.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2026-02-22T21:45:26.364 INFO:teuthology.orchestra.run.trial125.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-22T21:45:26.366 INFO:teuthology.orchestra.run.trial125.stderr: 2026-02-22T21:45:26.366 INFO:teuthology.orchestra.run.trial125.stderr:real 0m0.104s 2026-02-22T21:45:26.366 INFO:teuthology.orchestra.run.trial125.stderr:user 0m0.327s 2026-02-22T21:45:26.366 INFO:teuthology.orchestra.run.trial125.stderr:sys 0m0.039s 2026-02-22T21:45:26.367 INFO:teuthology.orchestra.run.trial068.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-22T21:45:26.367 INFO:teuthology.orchestra.run.trial140.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2026-02-22T21:45:26.368 INFO:teuthology.orchestra.run.trial068.stderr: 2026-02-22T21:45:26.368 INFO:teuthology.orchestra.run.trial068.stderr:real 0m0.117s 2026-02-22T21:45:26.368 INFO:teuthology.orchestra.run.trial068.stderr:user 0m0.324s 2026-02-22T21:45:26.368 INFO:teuthology.orchestra.run.trial068.stderr:sys 0m0.034s 2026-02-22T21:45:26.373 INFO:teuthology.orchestra.run.trial140.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2026-02-22T21:45:26.376 INFO:teuthology.orchestra.run.trial140.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2026-02-22T21:45:26.378 INFO:teuthology.orchestra.run.trial140.stderr: 2026-02-22T21:45:26.378 INFO:teuthology.orchestra.run.trial140.stderr:real 0m0.084s 2026-02-22T21:45:26.378 INFO:teuthology.orchestra.run.trial140.stderr:user 0m0.255s 2026-02-22T21:45:26.378 INFO:teuthology.orchestra.run.trial140.stderr:sys 0m0.037s 2026-02-22T21:45:26.447 INFO:teuthology.orchestra.run.trial051.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-22T21:45:26.449 INFO:teuthology.orchestra.run.trial051.stderr: 2026-02-22T21:45:26.449 INFO:teuthology.orchestra.run.trial051.stderr:real 0m0.199s 2026-02-22T21:45:26.449 INFO:teuthology.orchestra.run.trial051.stderr:user 0m0.507s 2026-02-22T21:45:26.449 INFO:teuthology.orchestra.run.trial051.stderr:sys 0m0.129s 2026-02-22T21:45:26.450 INFO:tasks.ceph:Archiving logs... 2026-02-22T21:45:26.450 DEBUG:teuthology.misc:Transferring archived files from trial051:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial051/log 2026-02-22T21:45:26.452 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T21:45:26.769 DEBUG:teuthology.misc:Transferring archived files from trial068:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial068/log 2026-02-22T21:45:26.770 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T21:45:26.848 DEBUG:teuthology.misc:Transferring archived files from trial125:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial125/log 2026-02-22T21:45:26.849 DEBUG:teuthology.orchestra.run.trial125:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T21:45:26.926 DEBUG:teuthology.misc:Transferring archived files from trial140:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial140/log 2026-02-22T21:45:26.927 DEBUG:teuthology.orchestra.run.trial140:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T21:45:27.004 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 1922, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 1484, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph_manager.py", line 3303, in wait_until_healthy assert time.time() - start < timeout, \ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: timeout expired in wait_until_healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 1919, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 46, in nested if exit(*exc): ^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable malloc malloc _dl_new_object 2026-02-22T21:45:27.006 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T21:45:27.029 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T21:45:27.043 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T21:45:27.043 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T21:45:27.046 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T21:45:27.048 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T21:45:27.050 DEBUG:teuthology.orchestra.run.trial140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T21:45:27.332 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T21:45:27.343 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T21:45:27.343 DEBUG:teuthology.orchestra.run.trial051:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:45:27.344 DEBUG:teuthology.orchestra.run.trial068:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:45:27.346 DEBUG:teuthology.orchestra.run.trial125:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:45:27.348 DEBUG:teuthology.orchestra.run.trial140:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:45:27.359 INFO:teuthology.orchestra.run.trial051.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:45:27.361 INFO:teuthology.orchestra.run.trial068.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:45:27.362 INFO:teuthology.orchestra.run.trial125.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:45:27.364 INFO:teuthology.orchestra.run.trial140.stderr:bash: line 1: ntpq: command not found 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:=============================================================================== 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:^* 50.205.57.38 1 6 377 119 +47us[ +67us] +/- 4749us 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:^- 73.185.182.209 2 7 377 116 +3490us[+3490us] +/- 57ms 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:^- 149.28.200.179.vultruser> 2 7 377 115 -567us[ -567us] +/- 41ms 2026-02-22T21:45:27.375 INFO:teuthology.orchestra.run.trial051.stdout:^- 96-19-94-82.cpe.sparklig> 1 7 377 118 -752us[ -752us] +/- 32ms 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:=============================================================================== 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:^* time.lmtlabs.com 2 7 377 117 -100us[ -138us] +/- 15ms 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:^- ntp.speculation.org 2 7 377 117 -5960us[-5999us] +/- 63ms 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:^- 172-234-25-10.ip.linodeu> 2 6 377 52 -9706us[-9706us] +/- 74ms 2026-02-22T21:45:27.376 INFO:teuthology.orchestra.run.trial068.stdout:^- static.36.62.78.5.client> 4 6 377 51 +781us[ +781us] +/- 55ms 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:=============================================================================== 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:^- 172-232-15-202.ip.linode> 2 6 377 52 -8720us[-8755us] +/- 76ms 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:^+ vps5.drown.org 2 7 377 52 -2538us[-2573us] +/- 22ms 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:^* time.ecansol.net 1 6 377 49 +1537us[+1502us] +/- 15ms 2026-02-22T21:45:27.380 INFO:teuthology.orchestra.run.trial140.stdout:^- 172-235-154-118.ip.linod> 4 6 377 54 -3216us[-3250us] +/- 52ms 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:=============================================================================== 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:^+ 172-233-189-68.ip.linode> 2 7 377 115 -3541us[-3541us] +/- 85ms 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:^* 44.190.5.123 2 6 377 116 +702us[ +583us] +/- 34ms 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:^+ white.web-ster.com 2 6 377 53 +2795us[+2795us] +/- 73ms 2026-02-22T21:45:27.384 INFO:teuthology.orchestra.run.trial125.stdout:^+ hou1.us.ntp.li 2 7 377 116 -703us[ -703us] +/- 71ms 2026-02-22T21:45:27.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T21:45:27.395 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T21:45:27.396 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T21:45:27.410 DEBUG:teuthology.orchestra.run.trial051:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-22T21:45:27.473 DEBUG:teuthology.orchestra.run.trial068:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-22T21:45:27.526 DEBUG:teuthology.orchestra.run.trial125:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-22T21:45:27.582 DEBUG:teuthology.orchestra.run.trial140:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-22T21:45:27.637 DEBUG:teuthology.orchestra.run.trial051:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:45:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:27.665 DEBUG:teuthology.orchestra.run.trial068:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:45:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:27.692 DEBUG:teuthology.orchestra.run.trial125:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:45:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:27.721 DEBUG:teuthology.orchestra.run.trial140:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T21:45:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:27.757 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T21:45:27.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T21:45:27.833 INFO:teuthology.task.internal:Duration was 1253.902161 seconds 2026-02-22T21:45:27.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T21:45:27.865 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T21:45:27.866 DEBUG:teuthology.orchestra.run.trial051:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T21:45:27.879 DEBUG:teuthology.orchestra.run.trial068:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T21:45:27.883 DEBUG:teuthology.orchestra.run.trial125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T21:45:27.887 DEBUG:teuthology.orchestra.run.trial140:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T21:45:27.914 INFO:teuthology.orchestra.run.trial051.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:45:27.919 INFO:teuthology.orchestra.run.trial068.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:45:27.921 INFO:teuthology.orchestra.run.trial125.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:45:27.934 INFO:teuthology.orchestra.run.trial140.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T21:45:28.431 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T21:45:28.431 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial051.front.sepia.ceph.com 2026-02-22T21:45:28.431 DEBUG:teuthology.orchestra.run.trial051:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T21:45:28.450 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial068.front.sepia.ceph.com 2026-02-22T21:45:28.450 DEBUG:teuthology.orchestra.run.trial068:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T21:45:28.490 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial125.front.sepia.ceph.com 2026-02-22T21:45:28.490 DEBUG:teuthology.orchestra.run.trial125:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T21:45:28.512 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial140.front.sepia.ceph.com 2026-02-22T21:45:28.512 DEBUG:teuthology.orchestra.run.trial140:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T21:45:28.533 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T21:45:28.534 DEBUG:teuthology.orchestra.run.trial051:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.537 DEBUG:teuthology.orchestra.run.trial068:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.541 DEBUG:teuthology.orchestra.run.trial125:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.554 DEBUG:teuthology.orchestra.run.trial140:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.635 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T21:45:28.635 DEBUG:teuthology.orchestra.run.trial051:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:28.638 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:28.640 DEBUG:teuthology.orchestra.run.trial125:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:28.643 DEBUG:teuthology.orchestra.run.trial140:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:45:28.658 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:45:28.659 INFO:teuthology.orchestra.run.trial051.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:45:28.659 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.659 INFO:teuthology.orchestra.run.trial051.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T21:45:28.659 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T21:45:28.660 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:45:28.661 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:45:28.662 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T21:45:28.662 INFO:teuthology.orchestra.run.trial068.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.662 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T21:45:28.663 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:45:28.664 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:45:28.664 INFO:teuthology.orchestra.run.trial125.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T21:45:28.664 INFO:teuthology.orchestra.run.trial125.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.664 INFO:teuthology.orchestra.run.trial125.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T21:45:28.666 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:45:28.667 INFO:teuthology.orchestra.run.trial140.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:45:28.667 INFO:teuthology.orchestra.run.trial140.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:45:28.667 INFO:teuthology.orchestra.run.trial140.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T21:45:28.668 INFO:teuthology.orchestra.run.trial140.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T21:45:28.669 INFO:teuthology.orchestra.run.trial051.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T21:45:28.672 INFO:teuthology.orchestra.run.trial068.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T21:45:28.674 INFO:teuthology.orchestra.run.trial125.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T21:45:28.677 INFO:teuthology.orchestra.run.trial140.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T21:45:28.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T21:45:28.695 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T21:45:28.696 DEBUG:teuthology.orchestra.run.trial051:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T21:45:28.735 DEBUG:teuthology.orchestra.run.trial068:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T21:45:28.761 DEBUG:teuthology.orchestra.run.trial125:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T21:45:28.787 DEBUG:teuthology.orchestra.run.trial140:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T21:45:28.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T21:45:28.829 DEBUG:teuthology.orchestra.run.trial051:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.832 DEBUG:teuthology.orchestra.run.trial068:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.835 DEBUG:teuthology.orchestra.run.trial125:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.838 DEBUG:teuthology.orchestra.run.trial140:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.855 INFO:teuthology.orchestra.run.trial051.stdout:kernel.core_pattern = core 2026-02-22T21:45:28.857 INFO:teuthology.orchestra.run.trial068.stdout:kernel.core_pattern = core 2026-02-22T21:45:28.859 INFO:teuthology.orchestra.run.trial125.stdout:kernel.core_pattern = core 2026-02-22T21:45:28.878 INFO:teuthology.orchestra.run.trial140.stdout:kernel.core_pattern = core 2026-02-22T21:45:28.892 DEBUG:teuthology.orchestra.run.trial051:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:28.929 DEBUG:teuthology.orchestra.run.trial068:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:28.946 DEBUG:teuthology.orchestra.run.trial125:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:28.963 DEBUG:teuthology.orchestra.run.trial140:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:45:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:45:28.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T21:45:28.998 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T21:45:28.999 DEBUG:teuthology.misc:Transferring archived files from trial051:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial051 2026-02-22T21:45:28.999 DEBUG:teuthology.orchestra.run.trial051:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T21:45:29.033 DEBUG:teuthology.misc:Transferring archived files from trial068:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial068 2026-02-22T21:45:29.033 DEBUG:teuthology.orchestra.run.trial068:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T21:45:29.067 DEBUG:teuthology.misc:Transferring archived files from trial125:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial125 2026-02-22T21:45:29.067 DEBUG:teuthology.orchestra.run.trial125:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T21:45:29.103 DEBUG:teuthology.misc:Transferring archived files from trial140:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64029/remote/trial140 2026-02-22T21:45:29.103 DEBUG:teuthology.orchestra.run.trial140:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T21:45:29.140 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T21:45:29.141 DEBUG:teuthology.orchestra.run.trial051:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T21:45:29.143 DEBUG:teuthology.orchestra.run.trial068:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T21:45:29.146 DEBUG:teuthology.orchestra.run.trial125:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T21:45:29.149 DEBUG:teuthology.orchestra.run.trial140:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T21:45:29.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T21:45:29.203 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T21:45:29.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T21:45:29.216 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T21:45:29.216 DEBUG:teuthology.orchestra.run.trial051:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T21:45:29.218 DEBUG:teuthology.orchestra.run.trial068:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T21:45:29.220 DEBUG:teuthology.orchestra.run.trial125:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T21:45:29.221 DEBUG:teuthology.orchestra.run.trial140:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T21:45:29.232 INFO:teuthology.orchestra.run.trial051.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 21:45 /home/ubuntu/cephtest 2026-02-22T21:45:29.235 INFO:teuthology.orchestra.run.trial068.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 21:45 /home/ubuntu/cephtest 2026-02-22T21:45:29.236 INFO:teuthology.orchestra.run.trial125.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 21:45 /home/ubuntu/cephtest 2026-02-22T21:45:29.247 INFO:teuthology.orchestra.run.trial140.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 21:45 /home/ubuntu/cephtest 2026-02-22T21:45:29.248 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T21:45:29.265 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T21:45:29.305 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/fixed-4 d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 1253.9021608829498 failure_reason: '"2026-02-22T21:37:28.298282+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)" in cluster log' flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-22T21:45:29.306 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T21:45:29.414 INFO:teuthology.run:FAIL